Interventions based on early intensive applied behaviour analysis for autistic children: a systematic review and cost-effectiveness analysis

Design of the research

A protocol was written and circulated to the members of the PPI group prior to the first of three Advisory Group meetings. This first meeting outlined the purpose and nature of the project and sought input on the study design, as a result of which we adapted our project design by:

• developing a list of relevant baseline characteristics that were important to include for the analyses

• listing which outcomes were important to relevant stakeholder groups

• further developing a definition of early intensive ABA-based treatments and narrowing the inclusion criteria

• further developing search terms to ensure that they were comprehensive.

Design of the analyses

The original intent of the second meeting of the Advisory Group was to focus on the design of the economic model to ensure that it reflected key elements of autism and had face validity. However, we extended the remit of this meeting to give an update on progress and outline the quality of the studies, the interventions used and the types of outcomes that we were able to collect. The discussions of this group formed the basis of both the structure of the eventual analyses and the economic model.

Analysis of the results

The PPI group and the other members of the Advisory Group were not involved at the analysis stage to retain a measure of independence owing to the strong beliefs that many members of the public and service providers already held about the effectiveness of the intervention. We did, however, include them in the interpretation of these results. These results were circulated prior to the final Advisory Group meeting, which discussed the potential interpretations, how far we should extend the economic model and what scenarios to include.

Reporting of the research

Patient and public involvement members, as part of the Advisory Group, had an opportunity to contribute to drafts of the final report before submission. They provided extensive comments on the document via e-mail.

External stakeholder consultation

After submission, relevant external organisations and groups were also consulted in a consultation exercise. When possible, we have amended the report based on these comments and in the interests of full transparency we have included the unaltered feedback from these individuals and organisations in Appendix 20. We also included the comments of one member of the PPI group in this section who was not able to provide feedback at an earlier stage. Full permission has been received from those who made these comments to reproduce them in this report.

Population

Studies that included children with a diagnosis of ASD, including any of the following terms: autism, Asperger syndrome, pervasive developmental disorder – not otherwise specified (PDD-NOS), atypical ASD or ASD1 based on Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, criteria (ICD-10). Inclusion was not restricted by age, although the primary focus of the review is on children of preschool age (aged < 5 years in the UK). If studies with a broader age were identified, preschool age children could be identified within the IPD.

Intervention

Studies of early intensive interventions based on ABA principles were eligible for inclusion in the review.

Intensive behavioural interventions:

• included > 15 hours per week of planned intervention

• used a comprehensive approach, targeting a range of behaviours, skills and developmental domains

• used ABA-based teaching strategies as the core components

• were delivered face to face by qualified or trained individuals

• were delivered at least initially on a one-to-one basis

• included qualified supervision of the therapists/trained staff/parents delivering the intervention.

Studies of interventions delivered to parents rather than children and studies of narrowly targeted interventions aimed at a single behaviour (e.g. joint attention) were excluded.

Comparators

Inclusion was not restricted by study comparator.

Relevant comparators included all other forms of early intervention, such as augmented forms of communication [e.g. the Picture Exchange Communication System (PECS)];55 other speech and language therapy interventions; support programmes led by independent providers, such as charitable and third-sector organisations; educational-based structured teaching approaches, such as Treatment and Education of Autistic and Communication related handicapped CHildren (TEACCH);56 and ‘eclectic’ approaches, as well as placebo, waiting list or TAU groups.

Outcomes

All studies that met the above criteria were included and, when available, contributed data for the prespecified IPD meta-analysis outcomes (see Chapter 6).

Study design

Prospective randomised and non-randomised controlled studies meeting all other inclusion criteria were eligible for inclusion. Non-comparative, single-arm studies were excluded.

Published data

Two researchers independently extracted data from published and unpublished study reports. Data were extracted on study design, intervention and comparator characteristics, baseline characteristics and results. Any discrepancies were resolved by discussion and, when necessary, cross-referencing IPD or contacting study authors.

Individual participant data

Study investigators were invited to supply data in a standardised coding format developed for SCABARD. However, data were accepted in any reasonable format and recoded as necessary by the research team. Data were requested for all recruited children, including any who were excluded from original study analyses. Study protocols were also requested from authors.

When the intervention or its components were unclear from publicly available sources, study authors were asked to provide these details alongside their IPD.

The SCABARD data dictionary, listing all data items requested, is given in Appendix 3.

We requested that all participant names and identifying numbers were removed prior to supplying data.

Secure file transfer and encrypted e-mail were offered as secure methods for transferring IPD. All data were anonymous and held in a password-protected area of the University of York servers.

Critical appraisal of studies

Critical appraisal of included studies was based on assessment of trial publications, protocols (if available) and by checking received data sets.

Risk of bias in RCTs was assessed using the Cochrane Risk of Bias tool. 57 Non-randomised controlled study designs were assessed using the Cochrane Risk Of Bias In Non-randomised Studies – of Interventions (ROBINS-I) tool. 58

As well as assessing aspects of bias (such as blinding and/or independence of outcome assessors), when possible, the critical appraisal also assessed the fidelity of delivered interventions to the underlying treatment model. Assessment was undertaken independently by two researchers, with any discrepancies resolved by consensus or recourse to a third researcher, if necessary.

Checking and quality assurance of individual participant data

All IPD were checked on receipt. Baseline data were tabulated and compared with the study publication, with any inconsistencies noted. Data were checked for internal consistency and integrity of randomisation (if conducted). Patterns of missing data were examined. One researcher ran data checks, which were independently cross-checked by a second researcher. Findings of all data checking were discussed with senior members of the research team.

The impact these checks had on the potential quality of the studies and data was considered (e.g. whether or not there was evidence of non-random allocation). This was used alongside the critical appraisal and could upgrade or downgrade the findings of the overall quality assessment.

Comparison of data available from published and individual participant data sources

The appraisal of included evidence was also informed by comparing the variables and outcomes collected in the IPD against those (1) requested by the SCABARD team (see Appendix 3), (2) reported in corresponding publications and (3) reported in original study protocols (if available).

Analysis framework and structure

Statistical details of individual participant data meta-analyses

Both one- and two-stage meta-analyses were performed for each outcome, provided that data were available for at least two studies. When these thresholds were not reached, a narrative summary of study results was produced.

Impact of covariates on treatment effect

Access to IPD means that the analysis can potentially go beyond looking only at whether or not early intensive ABA-based interventions are effective, to consider whether or not child-level characteristics (including parental and intervention factors specific to each child) might alter how effective the intervention is. For example, whether or not IQ at time of recruitment alters how effective EIBI is in changing outcomes. The impact the covariate may have on effectiveness is called the intervention–covariate interaction.

Time of measurement

Analyses were performed at 1 and 2 years after recruitment for each outcome. A tolerance of ± 6 months was used for each analysis. This means that, for example, measurements made from between 18 and 30 months could contribute to analyses at 2 years.

In a few studies, IPD were provided at times other than 1 or 2 years. To incorporate those additional data captured at other times, repeated measures analyses were performed. Repeated measures models analyse all time points simultaneously, so there is a single model estimating effects for all reported years. They also account for the fact that each child may have repeated measurements of the same outcome over time, which are likely to be correlated, by including a correlation term for each child.

When the data permitted, exploratory analyses were performed, including an assessment of whether outcomes varied linearly or log-linearly over time (i.e. assuming a trend over time rather than separate analyses). The choice of these models depended on the results of the analyses at each specific time point.

Studies not supplying individual participant data

When studies identified as eligible for inclusion in the meta-analysis did not supply IPD to the SCABARD team, relevant outcome data were extracted from study publications. Data were extracted as means and standard deviations (SDs) in each study arm, as 2 × 2 tables (numbers of events and participants by arm) or as relative risks, odds ratios or MDs if full data were unavailable.

Mean differences or SMDs for each outcome measure were calculated from extracted data. These were combined with the results for each study estimated from the IPD in exploratory two-stage meta-analyses, following the same process as described in Statistical details of individual participant data meta-analyses.

Meta-analyses combining IPD with published data from studies not supplying IPD were treated as sensitivity analyses and used to assess whether or not there are any differences between studies that did not supply IPD and those that did.

Missing data

When a study did not examine or record an outcome measure or a covariate, the study was excluded from all relevant analyses.

If > 20% of participants in the IPD had no record for an outcome measure, a best- and worse-case analysis was planned as a sensitivity analysis. All included studies had < 20% of participants with missing outcome data (when the outcome was collected), so this analysis was not required.

Complete-case analysis (excluding all participants with missing covariate data) was used for all analyses. Imputation analyses were considered in the protocol as a way of handling missing covariates, but were not performed, given the limited number of covariate analyses that were feasible and because data were largely complete for the analyses performed.

Sensitivity analysis

Although a number of sensitivity analyses were identified in the statistical analysis plan, the limitations of the IPD meant that the only sensitivity analysis performed was one limited to an analysis of only UK-based studies.

Network meta-analysis

Network meta-analyses (NMAs) analysed all types of intervention and control simultaneously. The one-stage repeated measures meta-analysis models described above were extended to include multiple arms and incorporated random effects to account for heterogeneity. Potential network inconsistency was investigated by comparing NMA results with results from direct pairwise meta-analyses.

Multivariate meta-analysis

The analysis included many outcomes that are likely to be highly correlated both within domains (e.g. different IQ scoring methods) and between domains (e.g. VABS score and autism symptom severity). Multivariate analysis of these correlated outcomes may improve estimation, particularly in cases in which some studies do not report one outcome, but do report a correlated outcome.

One-stage models of multivariate analysis were considered. Given the limited availability of outcomes, only bivariate analyses of composite VABS score with each other outcome were feasible. These analyses were done but are not reported here, owing to uncertainty as to their validity, given data limitations, and little evidence of any difference from the main univariate analyses.

Software

All data management and meta-analyses were performed at the Centre for Reviews and Dissemination, using the R software package (2016; The R Foundation for Statistical Computing, Vienna, Austria).

Additional libraries in R were used as follows:

• data management and manipulation: tidyr, dplyr, tidyverse libraries

• two-stage analyses: meta and metafor libraries

• one-stage models: lme4 library

• forest plots: using in-house R code and meta library

• other graphics: ggplot2 library.

Applied behaviour analysis-based early intensive intervention versus treatment as usual and eclectic comparators

Fifteen studies (14 published26,82–87,89–101,103–112 and one unpublished88) compared some form of EIBI with a TAU or eclectic intervention.

Applied behaviour analysis-based early intensive interventions of different intensity

Three studies compared EIBI with a lower-intensity variation of the same approach, requiring fewer one-to-one contact hours between child and therapist. 26,90,91,98,103,111 The original UCLA study by Lovaas26 compared 40 hours per week of EIBI against the same kind of treatment for < 10 hours per week (this study also included a retrospective cohort of children not receiving any ABA-based treatment, but insufficient data were available in either publications or IPD to inform the current meta-analysis). 26,98 A later US study compared 30 hours per week of planned intensive EIBI (plus 5 hours/week of parental treatment) against parental training in EIBI techniques alone. 103 One UK study compared 20–40 planned hours of ABA-based intervention with around 10–20 hours per week of the same approach. 90

Clinic- versus parent-directed applied behaviour analysis-based early intensive interventions

Two studies compared clinic-directed EIBI against some form of parent-directed EIBI. 99,100 In these studies, children in both treatment arms received similarly intensive intervention with therapists (30–40 hours/week), but the parent-directed groups either received less frequent supervision by senior therapists and clinical supervisors,100 or required parents to recruit and manage therapists. 99

Different forms of applied behaviour analysis-based early intensive behavioural therapy

One Canadian study, including 28 children, compared two forms of ABA-based intervention in young autistic children. 101 One treatment arm consisted of ‘Nova Scotia EIBI’, in which children received 15 hours per week of one-to-one instruction based on the PRT approach (a NDBI approach that targets ‘pivotal’ areas of a child’s development and emphasises natural reinforcement). The comparator arm was a group ABA preschool programme, based on the verbal behaviour method (a structured approach focused on teaching communication and language). Children in the group ABA group received 15–25 hours per week of training, of which 3–5 hours consisted of one-to-one discrete trial training.

Characteristics of early intensive applied behaviour analysis-based interventions

All included studies evaluated some form of early intensive ABA-based intervention. All such interventions were rooted in ABA and incorporated replications, extensions, adaptations or variations of teaching techniques originally described by Lovaas et al. at the UCLA during the 1970s and 1980s. 26,98 Early studies closely resembled the original UCLA method, although without physical aversives. 82,103 Subsequent studies have incorporated additional manualised ABA procedures into the original UCLA EIBI intervention model. 85,86,90,91,94,95,97

Several studies incorporated some or all the aspects of NDBI approach into the EIBI model. 83,84,88,92,93,99,100,104,106,107,110,112 This included approaches such as the ESDM. 83,93,106,107,112

Children in the included studies received these early intensive ABA-based interventions for a period of 9–36 months, at a planned intensity of 15–40 hours per week of mostly one-to-one teaching (when recorded). Comparator treatments were delivered for a similar duration, although treatment intensity was more variable, ranging from 2 to ≥ 30 hours per week (when recorded), with considerably less one-to-one contact.

Characteristics of comparator interventions

As stated in Chapter 4, Individual participant data meta-analysis methods, comparator treatments could be broadly classified as ‘eclectic’ or TAU. Comparators were classified as ‘eclectic’ when individual children were known to have received a mix of teaching approaches, such as TEACCH, PECS, other behavioural or development programmes, speech and language therapy, music therapy or occupational therapy. Ten studies included eclectic comparator intervention arms, eight of which were delivered in a school or nursery classroom setting84–86,89–93,95,96,105,108,110 and two of which were delivered in a university or specialist centre setting. 94,97 As well as an eclectic arm, one study included a portage treatment arm [a home-visiting educational service for preschool children with special educational needs (SEN) and disability]. 90,91

Other comparators were classified as TAU when children received non-autism-specific special education or other forms of standard local provision. Six studies included TAU arms, of which one was delivered in a school/nursery,105,108 three were delivered in a range of settings83,92,104,106,107,110,112 and two did not provide clear information about setting. 82,88 Study investigators were not typically involved in the provision of TAU comparator treatments, so often did not have detailed information on the exact interventions received by individual children (see Study quality and risk of bias).

Bias due to a range of confounding factors

All of the non-randomised studies were at serious risk of bias due to confounding. In five studies, the type of treatment received by children was explicitly based on parental preference,82,87,92,104,105,108,110 with parents actively seeking or lobbying for early intensive ABA-based treatment and, in some cases, paying for it themselves. 87,92,110

In other studies, the type of treatment received was primarily based on location84,86,90,91 or staff availability,26,85,95,98 for which the influence of parental preference was unclear.

In some studies, baseline differences in parental education, family composition or socioeconomic status were observed between treatment groups. 89,92,96,104,105,108–110

Bias due to deviation from intended interventions

Some studies described the methods used to assess treatment fidelity in the early intensive ABA-based intervention arm. These included monitoring, observation and feedback to tutors,83,93,104–108,112 or obtaining congruent descriptions of the intervention from parents and supervisors. 90,91 In other studies, early intensive ABA-based interventions were supervised, but without explicit monitoring for treatment fidelity. Studies noted difficulties, such as high tutor turnover resulting in the intervention being delivered for fewer hours per week than intended,92,110 high proportions of children not completing the intervention,104 unreliable recording of weekly hours of EIBI,84 and families changing between different EIBI organisations or supervisors and consultants during the study period. 89,96,109

The delivery and content of comparator arms was not closely monitored in the available studies, although three studies84–86,95 did report a high proportion of children receiving ABA techniques as part of ‘eclectic’ therapy or TAU comparators.

Studies rarely recorded whether or not children received any co-interventions alongside those being evaluated. One study appeared to compare groups in terms of independently procured co-interventions, finding that children receiving EIBI received more dietary and other biological interventions, extracurricular educational interventions and alternative treatments, than children receiving TAU. 89,96,109

Bias in measurement of outcomes

Truly independent and blinded measurement of outcomes were rarely achieved in the evaluation of early intensive ABA-based intervention studies. In some cases, the participants in the ABA-based intervention and the comparison intervention arm were assessed by treatment supervisors84 or study investigators,86 sometimes with an independent second evaluator. Although some studies described employing outcome assessors who were independent of direct intervention delivery,82,85,89,92,95,96,104,105,108–110 the assessments typically involved interaction with children and parents who were not blinded to intervention. Assessors who were blinded to allocation could potentially have been unblinded by the assessment location (if this differed between intervention arms or – when delivered in the family home – provided contextual information about likely treatment allocation). Consequently, all of the included studies were considered to be at moderate or serious risk of bias for this domain.

Summary

Although randomisation is clearly feasible, most studies used convenience samples, with the allocation to early intensive ABA-based interventions being based on location or parental preference. Although some attempts were made to avoid bias in the measurement of outcomes, the nature of the intervention can make true blinded assessment difficult to achieve. There is evidence from some studies to suggest differences between the two intervention groups in terms of socioeconomic status and use of co-interventions, but this information was not consistently recorded across studies. In some cases, outcome data were missing or available for only one treatment group. It is also important to note that despite requesting them, we did not receive any protocols for the included studies. The original Lovaas study,26 in particular, was at risk of several forms of bias, including the comparator groups differing on the few available baseline variables. 26,98 Taken together, these concerns increase our uncertainty about the results observed in several included studies, making it possible that the effects observed in the meta-analysis may overestimate the true effects of early intensive ABA-based interventions.

Interventions

Comparators

Comparator interventions for autistic children have also evolved over time, with the emergence of autism-specific rather than generic special needs care. More recently, ‘eclectic’ comparators have explicitly incorporated some ABA techniques. 83,84,86,89,93,94,96,97,105,108,109 In most studies, children in the eclectic or TAU comparator arms received fewer hours of intervention and/or less one-to-one contact. However, this was not always the case: Howard et al. 105,108 compared EIBI against an eclectic autism-specific classroom, with ≈30 hours per week of one-to-one or one-to-two intensive intervention; and Zachor et al. 94,97 reported both intervention groups receiving similar levels of funding per child, hours in preschool setting, support for parents and staff, and individual one-to-one treatments.

Although we did not restrict inclusion by comparator, all of the identified comparators were eclectic intervention or TAU; no studies compared early intensive ABA-based interventions with discrete medical or educational interventions.

It is difficult to map the comparators in the available evidence to the current UK standard provision for two reasons. First, most studies had very limited available information on the content of eclectic interventions or TAU, as the study investigators were rarely involved in their delivery. Second, there is evidence that standard provision in the UK differs substantially between local authorities, although information obtained from York local authority suggests a mix of therapies not dissimilar to those cited in the studies (Ruth Horner, City of York Council, York, 2018, personal communication).

Participants

The studies included in this review cover a period when a large increase in the annual incidence of autism has been observed (more than fivefold from 1988 to 1995). 113,114 Part of that increase has been attributed to changing and broadening diagnostic criteria,115,116 as well as increased medical and public awareness. 117 This raises concerns about whether or not children receiving early intervention ABA-based interventions in included studies are similar to those currently eligible for intervention in the UK.

The overall population of children for whom IPD were provided were young (mean age 38 months) with mild-to-moderate intellectual disability (mean IQ 57) (see Chapter 6, Individual participant data received from included studies). As all children had to have an established diagnosis of autism or related condition to participate in the included studies, the study populations ought to be comparable to those who would be eligible for treatment in the current UK context. However, some of the more highly controlled studies excluded children with comorbidities, so it is plausible that typical UK treatment populations are more heterogeneous than those in the available evidence.

Outcomes

Although we accepted any outcomes for the IPD meta-analysis, only a small number of outcome domains were consistently collected across the included studies (most commonly, verbal and non-verbal IQ, adaptive behaviour and language measures; less commonly, autism symptom severity, behaviours that challenge and school placement).

The original Lovaas study26,98 was almost entirely focused on IQ and mainstream schooling placement as measures of treatment success. Children who achieved IQ in the average range were considered ‘recovered’ and the authors made ‘considerable effort’ to keep these children in mainstream preschool. In some cases, this involved withholding the child's diagnosis of autism. The authors stated, ‘If the child became known as autistic (or as “a very difficult child”) during the first year in pre school, the child was encouraged to enrol in another, unfamiliar school (to start fresh)’. 26 Apart from any ethics and bias concerns it may raise, this excerpt shows how different the goals of early autism interventions were 30–40 years ago.

Subsequent studies incorporated behavioural measures, such as adaptive behaviour, while retaining IQ or cognitive development measures. Schooling as an outcome was only collected in a minority of studies and did not use consistent classifications. 26,89,92,96,98,103,104,109,110

Outcomes relating to social participation, well-being and quality of life were not measured among the included studies. Although measures of cognitive development and adaptive behaviour can be used to track progress and development, no studies investigated how these measures correlate with measures of well-being, either during treatment or in the long term. In fact, any measurement of outcome beyond the end of the early intervention treatment period was rare.

With the exception of one study stating that ‘no serious adverse effects related to the intervention were reported during the 2-year period’,83,106,107,112 adverse or unintended effects of intervention were not addressed in the available evidence, with no study providing IPD on adverse effects.

The selection of measurement tools used for the collected outcomes varied within and between studies. Different measures of IQ and cognitive development were used at baseline and follow-up, based on the relevant normative populations [e.g. the WPPSI-R67 or BSID scales at baseline and Wechsler Intelligence Scale for Children (WISC)65 at follow-up]. At baseline, in particular, decisions about which measure to use are also informed by children’s developmental skills and ability to meaningfully be assessed in particular tests. The IPD meta-analyses separate measures of non-verbal skills (such as the MPSMT)70 from standard intelligence tests, which include verbal and non-verbal scales, as the former are known to yield higher scores. 100

Several studies reported difficulties when using standardised measures in the evaluation of interventions in young autistic children, including floor and ceiling effects on different tests at different ages. Authors dealt with these difficulties by recording minimum or maximum scores,103 reporting age-equivalent scores,101 raw scores89,92,96,109,110 or the number of children capable of achieving a score. 92,110 Although the provision of IPD facilitated the harmonisation and synthesis of scores across some of these studies, this was not always possible or appropriate.

Most studies provided standard scores when these were available and these scores informed the IPD meta-analyses. However, although standard scores allow comparisons with typically developing populations, it has been argued that they may miss information about changes that are relevant within the autism population specifically. 118 This problem is not limited to early intensive ABA-based interventions, but is one of several barriers to valid and reliable outcome measurement in young autistic children in general. 119

Participant and study characteristics

Ten studies provided IPD for the comparison of early intensive ABA with TAU or eclectic interventions. 84–86,89–97,104,109,110 These studies reported a wide range of different eligible outcomes.

Appendix 7 shows which of the variables specified in the SCABARD data dictionary were available as IPD for each included study. The only individual-level variables that were available for every study were age at baseline, sex and assigned treatment arm. The majority of other variables were not consistently collected across studies. In some cases, study investigators clarified that these variables were not collected at all, collected for the early intensive ABA-based intervention arm only, or were otherwise unavailable. However, in most cases, variables were absent from the IPD data sets, without any definitive explanation being provided.

For a comparison of the outcome domains and measures in the IPD with those in the published literature, see Appendix 8.

A variety of scales were used to measure IQ and cognitive development, most commonly versions of the BSID62 S–B,69 WISC65 and Wechsler Preschool and Primary Scale of Intelligence (WPPSI). 67 When different scales were used in studies, this was typically due to the need to select an age-appropriate scale. As all these scales were standardised (to an average IQ of 100 with SD of 15), in the primary analysis of IQ we have not differentiated between the methods and consider them as equivalent.

Non-verbal intelligence was most commonly measured using the MPSMT70 and autism symptom severity with the ADOS. 78 Although adaptive behaviour was consistently measured using the VABS,59 language was measured using a variety of scales, most commonly the RDLS74 and the expressive and receptive subscales of MSEL. 76 Parental outcomes were not measured frequently or consistently across studies, and school placement IPD were available for only one study. 89,96,109

Data were provided at baseline and at least one follow-up time (as required for meta-analysis) for only a minority of outcomes. Table 3 summarises the outcomes recorded across studies. This shows that, for most outcomes and outcome domains, there were few studies and limited data for meta-analysis. For a comparison of the outcome domains and measures in the IPD against those in the published literature, see Appendix 8.

A further issue is that studies were not consistent in the times at which outcomes were recorded. Table 4 shows the number of studies reporting at each time for the outcomes listed in Table 3. This shows that, although several studies reported an outcome, not all studies reported it at consistent times. For example, several outcomes were reported only in one study, and only one study reported data at 7 years (precluding any meta-analysis).

Meta-analyses by outcome domains

This section presents results of meta-analyses for each outcome domain. Given the small numbers of studies involved, meta-analyses of any early intensive ABA-based intervention compared with TAU or eclectic interventions are presented.

Adaptive behaviour

Adaptive behaviour was assessed using the VABS composite and component scores, with results available for all 10 included studies. 84–86,89–97,104,109,110 Here, we consider the meta-analyses of the composite VABS score. The complete data on the composite score are shown in Figure 3. The points show the scores for each child, with the lines connecting each child’s score over time. Figure 4 shows the mean (average) change in score from baseline.

FIGURE 3.

Reported VABS composite scores for each child, by study and intervention.

FIGURE 4.

Mean change in VABS composite score from baseline, by study and intervention.

These plots suggest no consistent trend in scores across studies. Indeed, there is considerable variation in how VABS changes over time. In some studies, scores remained largely unchanged over time, in some there was little change in the control arm but increases in early intensive ABA-based interventions, and in some the scores decreased in both arms.

The meta-analyses of VABS composite score at 1 and 2 years are shown in Figures 5 and 6, respectively. These showed no clear evidence of a benefit of early intensive ABA-based interventions after 1 year, with substantial heterogeneity (MD 1.82, 95% CI –2.79 to 6.43, I² = 80%), and a 7-point difference in favour of the ABA-based intervention (half of a SD) after 2 years, with less heterogeneity (MD 7.74, 95% CI 1.87 to 13.61, I² = 72%). Studies varied substantially in their estimated MDs. One extreme outlier, with seven children, found a 32-point difference in favour of early intensive ABA-based interventions. In the opposite direction, one trial found a 5-point difference in favour of TAU or eclectic interventions.

FIGURE 5.

Meta-analysis of composite VABS score at 1 year.

FIGURE 6.

Meta-analysis of composite VABS score at 2 years.

Data beyond 2 years were limited. Only two small studies95,104 reported outcomes at 3 years and one study at 4 years,110 and so forest plots are not presented here. The data at 7 years came from only one study,96 which found no difference between groups at 7 years (MD –0.99, 95% CI –12.80 to 10.83).

Cognitive ability

Intelligence quotient (measured using any scoring method) was reported in the majority of studies. 85,86,89–92,94,96,104,109,110 We analysed IQ scores whatever assessment method was used.

Figure 7 shows the mean change from baseline in IQ, by study and intervention. As for VABS score, the pattern of changes varied across trials. Some trials had IQ improvements with both early intensive ABA-based and comparator groups, some showed improvement only with the early intensive ABA-based interventions and some showed no improvement at all.

FIGURE 7.

Mean change in IQ score from baseline, by study and intervention.

The meta-analyses of IQ at 1 and 2 years are shown in Figures 8 and 9, respectively. These show clear evidence of a difference in favour of early intensive ABA-based interventions of around 10 IQ points after 1 year (two-thirds of a SD, MD 10.12, 95% CI 5.81 to 14.44, I² = 0) and after 2 years (MD 11.97, 95% CI 6.74 to 17.20, I² = 15%).

FIGURE 8.

Meta-analysis of IQ at 1 year.

FIGURE 9.

Meta-analysis of IQ at 2 years.

Data beyond 2 years were limited. Only two small studies104 reported outcomes at 3 years and one reported outcomes at 4 years,92,110 and so forest plots are not presented here. The data at 7 years comes from only one study,96 which found no evidence of a difference between groups at 7 years (MD –1.92, 95% CI –15.17 to 11.32).

Meta-analyses of all outcome measures

This section considers the meta-analysis of all reported outcomes for which there were sufficient data for a meta-analysis comparing any early intensive ABA-based interventions with TAU or eclectic interventions (see Tables 2 and 4). This includes meta-analyses of subscales, such as the subscales of VABS.

Given the large number of analyses, only summary results from the one-stage repeated measures meta-analyses are presented here. Forest plots for all meta-analyses are available on request.

Figure 10 presents a summary of the results of the one-stage repeated measures meta-analyses. Each subplot represents a separate outcome scale (e.g. ADOS repetitive behaviours in the top left). The vertical axis shows the MD in effect between early intensive ABA-based interventions and TAU or eclectic interventions, and the horizontal axis indicates the year after recruitment at which each meta-analysis was performed. Each dot is the summary MD in effect between the interventions from the relevant one-stage analysis of variance meta-analysis, with the vertical line being its 95% CI. Note that each dot and line represents an independent meta-analysis for each year; no time trends are assumed or modelled here.

FIGURE 10.

Repeated measures meta-analysis of all outcomes. a, Reduction in ADOS score is considered a positive result. For all other scales, increased scores are considered positive. DLS, daily living skills.

A positive result (above the blue line) indicates results favouring the early intensive ABA-based intervention, except for ADOS scores, for which results below the line favour the early intensive ABA-based intervention.

The results for VABS composite score, IQ and ADOS severity score in Figure 10 are broadly similar to those shown in the forest plots in Meta-analyses by outcome domains (see Figures 5, 6, 8, 9 and 24).

For IQ 1 year after follow-up, the meta-analysis favoured the early intensive ABA-based intervention, with a MD between groups of around 9 points (MD 9.16, 95% CI 4.38 to 13.93). After 2 years, this increased to a 134-point difference (MD 14.13, 95% CI 9.16 to 19.10).

Results after 7 years are presented here for ease of reference, but are based on only one study. 17 This study found no statistically significant evidence of a difference between interventions at 7 years (MD 4.39, 95% CI –8.17 to 16.95).

The meta-analysis results for non-verbal IQ measured using the MPSMT approach were broadly similar to general IQ at both 1 year (MD 9.45, 95% CI 0.33 to 18.59) and 2 years (MD 10.13, 95% CI 1.58 to 18.68), with MDs between groups of around 10 points after 2 years.

Composite VABS score showed no clear evidence of a difference between groups at 1 year (MD 2.93, 95% CI –1.90 to 7.76) and a statistically significant difference of around 7 points in favour of the early intensive ABA-based intervention after 2 years (MD 7.00, 95% CI 1.95 to 12.06). VABS subscale scores followed the same pattern. Socialisation and daily living scores showed no clear evidence of a difference at 1 year, and about a 5- to 8-point difference in favour of early intensive ABA-based interventions at 2 years. Communication showed a 6- to 8-point difference in favour of early intensive ABA-based interventions after 1 and 2 years (at 2 years: MD 8.03, 95% CI 1.11 to 14.97). Results for motor skills were inconclusive. The single study17 reporting data after 7 years found no difference between interventions.

For ADOS severity score, there was a MD of about 0.4 points between arms, with results favouring TAU or eclectic interventions at 1 year, but this was not statistically significant (MD 0.43, 95% CI –0.23 to 1.08). Results at 2 years were based on a single study, but showed a non-significant MD of about 1 point between arms, with results favouring early intensive ABA-based interventions (MD –1.33, 95% CI –2.81 to 0.14). There were similar results for the repetitive behaviour and social subscales.

For language development, RDLS comprehension scores after 1 year had a MD of about 12 points between arms, with results favouring early intensive ABA-based interventions (MD 12.96, 95% CI 2.01 to 23.91) and similarly at 2 years (MD 11.78, 95% CI 2.12 to 21.45). Results were similar for expressive language, although these did not achieve statistical significance. By contrast, MSEL receptive and expressive language subscales showed no evidence of any difference between early intensive ABA-based interventions and control arms after 1 or 2 years. Similarly, there was no evidence of any effect of early intensive ABA when assessed using composite MSEL score, or its fine motor or visual reception subscales.

The figure includes some estimates of effect at 3 and 4 years. These are derived from limited data from three studies,95,104,110 so should be treated with caution, but are generally consistent with effect estimates at other times, with similar estimated MDs. The exception is a large effect on non-verbal IQ (MPSMT) after 4 years, but this is based on one study110 with very few children.

Impact of child-level covariates

As described in the methods section (see Chapter 4, Impact of covariates on treatment effect), we intended to explore a range of child- and parent-level covariates. Unfortunately, the IPD supplied from most studies were too limited to permit investigation of child-level covariates. The only covariates with sufficient data for analysis were age and sex. To these covariates, we added baseline IQ and baseline composite VABS score.

To assess whether or not these covariates alter the effectiveness of early intensive ABA-based interventions, the repeated measures models were extended to include the relevant covariates and the interaction between covariate and early intensive ABA-based interventions, as set out in Chapter 4, One-stage analyses. It should be noted that these analyses test whether or not the MD between interventions varies with the covariate of interest (e.g. is the difference between VABS scores using early intensive ABA-based interventions compared with using TAU or eclectic interventions larger in girls than in boys?). We are not investigating whether or not the actual values of the outcomes vary with the covariates (e.g. if children with higher initial IQ have greater improvements in IQ or VABS after intervention). The results of these analyses for the outcomes IQ and composite VABS score (where there are most data) are shown in Table 5. Full results from all outcomes are given in Appendix 11 (see Table 45). The main point to note from these results is that CIs are very wide, spanning no effect (a value of zero), and so all p-values are well above 0.05. Therefore, there is no clear evidence of any interaction between these factors and either IQ or VABS score (e.g. no evidence that older children gain greater benefit from early intensive ABA-based interventions than with alternative interventions than younger children).

Impact of study-level covariates

A number of study-level covariates were investigated to determine if they had any impact on the effectiveness of early intensive ABA-based interventions. Owing to the large numbers of possible analyses, we considered only two outcomes, IQ and composite VABS score, at 1 and 2 years after recruitment, as these were the only outcomes with sufficient studies for reliable analysis.

Categorical covariates were assessed using subgroup analyses. Factors considered were:

• allocation method (parental choice, location based, cohort)

• delivery setting (home, school, specialist centre)

• parental involvement in ABA (none, encouraged, some)

• use of ABA methods in control intervention (none, partial).

Table 6 presents a summary of these subgroup analyses for outcomes at 2 years. For IQ, all p-values are > 0.05 and there are no clear patterns of variation in treatment effect across analyses. For VABS composite score, all p-values are < 0.1, but these are a consequence of extreme results in a single subgroup, driven by extreme results in a single trial,85,95 and are unlikely to represent genuine differences between subgroups.

Continuously distributed covariates were assessed using meta-regression. Covariates considered were:

• planned duration of early intensive ABA-based intervention

• planned intensity (hours/week) of early intensive ABA-based intervention

• year of study publication (to test for variation in effects over time).

Table 7 presents the results of the meta-regression analyses at 2 years. There is some suggestion that older studies had larger effects on the VABS composite scale than more recent studies (by 1.82 points on the VABS scale/year). This might suggest biased results in early studies, or that changes in how early intensive ABA-based interventions are given has led to changes in effectiveness. There was no evidence to support an impact of any other of the covariates on the effectiveness of early intensive ABA-based interventions. It should, however, be remembered that these analyses are based on few studies, and may be prone to bias.

Sensitivity analyses

The main analyses combined all forms of high-intensity early intensive ABA-based interventions compared with TAU or eclectic interventions. Subgroup analyses were performed to compare studies using TAU as a control with those using eclectic treatments, and to compare studies using EIBI only (e.g. Lovaas/UCLA model) with those using EIBI with NDBI. A summary of the results for IQ and VABS composite score at 2 years is shown in Table 8.

There was no clear evidence that the type of intervention or the type of control used leads to different results. In theory, results may be confounded as studies using only EIBI were generally compared with ‘eclectic’ control, whereas studies using EIBI with NDBI were generally compared with TAU. However, as noted previously, these comparator labels can be used somewhat interchangeably.

UK-based studies

Three of the included studies were conducted in the UK. 89–92,96,109,110 To assess whether or not the performance of early intensive ABA-based interventions in the UK was consistent with the overall meta-analysis, we performed a sensitivity analysis in the UK-based studies, repeating the main one-stage repeated measures analysis from Figure 10. The results of this analysis are shown in Figure 11, restricted to IQ and VABS subscales, as there were insufficient data for analysis of other outcomes.

FIGURE 11.

Repeated measures meta-analysis of the three UK-based studies. DLS, daily living skills.

The results suggest slightly smaller effects in the UK-based studies, particularly for composite VABS score, for which the MD between early intensive ABA-based and comparator arms at 2 years was 2.78 (95% CI –1.98 to 7.54), compared with 6.33 in the main analysis. For IQ, at 2 years the MD was 9.34 (95% CI 0.91 to 17.86) in the UK studies, compared with 13.64 in the main analysis. The smaller sample size also means that CIs were wider and results were not generally statistically significant.

Longer-term data at 4 and 7 years were reported in only one study in each case. At 7 years there was no evidence of any difference between early intensive ABA-based intervention and control, as discussed in Adaptive behaviour and Cognitive ability, with most effect estimates close to 1 and no statistically significant results. 96 At 4 years, there was similarly no evidence of any difference, except for VABS daily living score (MD 9.89, 95% CI 0.87 to 18.91). 110

However, there is insufficient evidence to assert that UK studies are different from others, and the reasons for observed differences may be varied and not due to study location.

Inclusion of studies not providing individual participant data

Of the five studies that did not provide IPD, four have been published82,83,87,105–108 and collected a similar variety of outcome domains and measures as studies that provided IPD. Additional outcomes included measures of repetitive behaviours, shared positive affect, gesture use, and the visual reception and fine motor subscales of the MSEL (see Appendix 8).

Two of the studies did not report outcome data in a form that could be used in the meta-analysis. 82,87 Two studies comparing high-intensity ABA-based interventions with comparator interventions have published usable outcome data. 83,105–108,112 Included also is one study that was published after IPD collection had closed. 102 To investigate the impact of these studies on the meta-analyses we extracted data from their publications and combined them with the IPD to perform two-stage change-from-baseline meta-analyses (in which the mean change in outcome between observation time and baseline is analysed). The full results are presented in Appendix 11 (see Figures 27–32 and Table 42). Figures 12 and 13 show the forest plots for IQ and composite VABS score at 2 years. These forest plots present sensitivity meta-analyses, in which the two studies with data extracted from publications are analysed with studies that provided IPD.

FIGURE 12.

Intelligence quotient at 2 years, including studies not providing IPD.

FIGURE 13.

Composite VABS score at 2 years, including studies not providing IPD.

Both analyses produce larger differences between the two groups in favour of early intensive ABA-based interventions than observed in the main analyses using only IPD (see Figures 6 and 9). This is because of the very large effects found by Howard et al.,105 which are approximately double those estimated from the IPD meta-analysis. The other studies that did not provide IPD83,102 had results more consistent with the IPD meta-analysis.

Narrative synthesis of other outcomes

Meta-analyses were performed for all outcomes reported by at least two studies at a common follow-up time. There were a number of other outcomes reported in only one study, or in two studies at inconsistent times. The results of these further outcomes are summarised in Appendix 11 (see Table 43).

Data on these outcomes are too limited to draw any meaningful conclusions. However, most results were in the direction of favouring early intensive ABA-based interventions, suggesting small reductions in autism symptom severity, behaviours that challenge, and improvements in IQ and language development. These are consistent with the meta-analyses of other outcomes.

High- versus low-intensity early intensive behavioural intervention

Three studies compared high-intensity EIBI (> 15 hours/week) with lower-intensity EIBI. 26,90,91,98,103 IQ was the only outcome recorded consistently in all three studies. A repeated measures meta-analysis of IQ comparing high- with low-intensity early ABA-based intervention is shown in Figure 14. This suggests that high-intensity ABA-based interventions produced much larger improvements in IQ than low-intensity ABA-based interventions, with differences of between 10 and 20 points from 1 to 4 years after recruitment, although results are not statistically significant at 1 and 3 years.

FIGURE 14.

Mean difference in IQ between high- and low-intensity EIBI. DLS, daily living skills.

Parental-managed compared with other applied behaviour analysis

Two studies compared parent-directed or managed early intensive ABA-based interventions (specifically EIBI with NDBI) with clinician or other management of the same intervention. 99,100 The repeated measures meta-analysis of these two studies is summarised in Figure 15. There is no indication of difference in any outcome, with all CIs including no effect and effect estimates are inconsistent in the direction of effect. There is therefore little evidence based on these data that parental management of early intensive ABA-based intervention alters its effectiveness.

FIGURE 15.

Meta-analysis of parent managed compared with other management of ABA. DLS, daily living skills.

Other comparisons

One study (28 participants) specifically compared two forms of ABA-based early intervention: PRT-based intervention (Nova Scotia EIBI) compared with a group-based verbal behaviour intervention (group ABA).

Over 1 year, results for cognitive development, receptive and expressive language, adaptive behaviour, problem behaviour and parenting stress, all indicated small and statistically non-significant differences between the two treatment arms (see Table 41 in Appendix 10).

School placement

Three studies provided IPD on the school placement of children. 26,89,96,98,103,109 Only the study by Magiati et al. compared early intensive ABA-based interventions with TAU or eclectic interventions;89,96,109 the other two studies compared high- with low-intensity EIBI. 26,98,103 To analyse school placement we considered low-intensity EIBI to be equivalent to eclectic treatment (see Network meta-analysis for the justification for this).

The results suggest that generally only children receiving high-intensity EIBI proceed to mainstream education. Children receiving low-intensity or eclectic interventions are more likely to go to specialist schools.

The regression model found very large and statistically significant odds in favour of going to more mainstream education with high-intensity intervention. These odds appear implausibly large. Further regression models adjusting for IQ and VABS composite score (results not shown) found that, even after adjusting for these factors, there were substantial odds in favour of going to more mainstream education with high-intensity intervention. This suggests that school placement is not dependent on improvements in IQ or behaviour. So difference in school placement could potentially be driven by knowledge of the intervention received and by parental selection, values, preferences and priorities, among other factors. Therefore, it is not clear whether or not high-intensity ABA-based intervention improves chances of going to a mainstream school.

Figure 16 shows the distribution of school placement by study and intervention arm. Figure 17 shows the result of a proportional odds regression to estimate the odds of being placed in a ‘more mainstream’ school if receiving high-intensity ABA-based intervention. The results suggest that generally only children receiving high-intensity ABA-based intervention proceed to mainstream education and children receiving low-intensity or eclectic interventions are more likely to go to specialist schools.

FIGURE 16.

School placement of children.

FIGURE 17.

Proportional odds regression for school placement.

High- compared with low-intensity intervention

Given the evidence from the NMA that, in general, all high-intensity interventions appear to have similar effectiveness (likewise, for all low-intensity interventions), we performed a further post hoc meta-analysis of all studies comparing any high-intensity intervention with any low-intensity intervention. This allowed the studies comparing high- and low-intensity ABA-based interventions to be meta-analysed along with those comparing high-intensity ABA-based interventions with alternative interventions, increasing the sample size.

The results of one-stage repeated measures meta-analyses are shown in Figure 19. They were generally very similar to the meta-analysis presented in Figure 10. The MD in IQ between high-intensity interventions and low-intensity interventions is around 17 points (at 2 years: MD 16.76, 95% CI 10.60 to 22.93). Communication, as measured by the VABS subscale, also favoured high-intensity intervention from 1 year (MD 8.18, 95% CI 0.54 to 15.85). For other VABS components, results favoured high-intensity ABA from 2 years onwards, with a MD compared with low-intensity ABA of around 5–8 points (VABS composite at 2 years: MD 8.60, 95% CI 2.86 to 14.34).

FIGURE 19.

Repeated measures meta-analysis of high- compared with low-intensity intervention studies. DLS, daily living skills.

As in the main meta-analyses (see Figure 10), results favoured high-intensity intervention for language comprehension measured using RDLS (comprehension at 2 years: MD 11.66, 95% CI 2.00 to 21.33). There was no evidence of any difference between high- and low-intensity intervention if using the MSEL expressive and receptive language subscales. There remains little evidence on autism symptom severity, but there was a suggestion (not statistically significant) that results favoured high-intensity intervention at 2 years (ADOS severity score at 3 years: MD –1.34, 95% CI –2.82 to 0.14).

Intelligence quotient and Vineland Adaptive Behaviour Scale

The main conclusion from all the meta-analyses is that receiving early high-intensity ABA-based intervention leads to somewhat greater improvements in IQ and VABS scores than if receiving low-intensity, TAU or eclectic interventions. IQ scores were higher by at least 9 points (about 0.6 of a SD) from 1 year after initiation of intervention onwards (MD 9.16, 95% CI 4.38 to 13.93). Scores on the VABS components were higher in the ABA intervention arms by 6–8 points on average (0.8 of a SD) after 2 years (composite score: MD 7.00, 95% CI 1.95 to 12.06). Improvements at 1 year appeared to be smaller and were not generally statistically significant.

Heterogeneity, estimated by I², was generally low, but this was a consequence of the small size of most included studies, which makes it difficult to conclusively identify heterogeneity. There was, however, considerable variation in effect estimates across studies for some outcome measures. For example, MDs in composite VABS at 2 years ranged from –4.79 to > 33. There was also variation across studies in how the outcome measures changed over time. This raises some concerns of whether or not studies are comparable.

There were few data on the longer-term impact of early intensive ABA-based interventions beyond the end of the main intervention period. Only one study reported results at 7 years after initiation, which found no difference between early intensive ABA-based and comparator arms. It is therefore not possible to determine the long-term impact of early intensive ABA-based interventions, including whether or not any benefits observed at 2 years persist.

Other outcomes

Results for other outcomes were limited due to the small numbers of studies reporting them. Data on autism symptom severity were too limited to be conclusive, with no clear evidence that early intensive ABA-based interventions improved the severity of autism compared with TAU or eclectic interventions. There were, similarly, limited data on language comprehension. Three studies85,89,104 that used the RDLS scale found some evidence that language scores favoured the early intensive ABA-based group compared with TAU or eclectic interventions by around 11 points (RDLS comprehension at 2 years: MD 11.78, 95% CI 2.12 to 21.44). By contrast, two studies93,97 that reported the MSEL expressive and receptive language subscales found no difference in scores between the two groups. There were insufficient data on behaviours that challenge to permit any meta-analysis. The limited data from one study90 suggested that early intensive ABA-based interventions could reduce behaviours that challenge, but results were not statistically significant.

School placement was available in only one study89 comparing early intensive ABA-based interventions with TAU and eclectic interventions, and in two studies26,103 comparing high- with low-intensity ABA-based interventions. These studies suggested that children receiving a high-intensity ABA-based intervention were much more likely to be in mainstream education (possibly with support) than children receiving TAU, eclectic or low-intensity ABA-based interventions. This analysis, however, may be confounded by parental preference for early intensive ABA-based treatment and mainstream schooling placement, and the practice in Lovaas26 of encouraging children to change school if teachers became aware of their autism. 26 Consequently, there is no good-quality evidence to suggest that early intensive ABA-based interventions independently influences school placement.

Participant and study characteristics

Studies reported limited data on child and parent characteristics, or details of how early intensive ABA-based interventions were administered. It was therefore not possible to investigate most of the protocol-specified covariates and how they might influence the effectiveness of early intensive ABA-based interventions. We found no conclusive evidence that age, sex, baseline IQ or baseline VABS score had any impact on the effectiveness of early intensive ABA-based interventions when compared with other interventions.

There was no generally clear evidence that study-level covariates, including delivery setting, allocation, parental involvement, duration and intensity of early intensive ABA-based treatment, had any clear or observable impact on the MDs between these interventions and comparator groups. A possible exception was that trials from earlier years found larger impacts on VABS composite score than later trials, although the power to detect such differences was limited as there were few studies in the analysis.

Comparison of applied behaviour analysis-based interventions

Sensitivity analyses and the NMA both found no evidence of any difference in effectiveness between TAU and eclectic interventions or any difference in effectiveness between different types of ABA-based early intervention (EIBI alone or with NDBI). There was no evidence that parent-managed interventions differed in effectiveness from other forms of management.

There was some evidence that high-intensity EIBI may be more effective than low-intensity EIBI, and that high-intensity ABA-based intervention, in general, may be more effective than low-intensity interventions (whether low-intensity ABA based or eclectic interventions). This may suggest that it is the high intensity of the intervention that causes the greater benefits, rather than the precise nature of the intervention received.

Scenario analysis exploring adult outcomes

Although the main (base-case) analysis limits the time horizon to childhood (15.5 years), an exploratory scenario analysis considered the potential impact of incorporating adult outcomes into the model. In this scenario, an additional phase is added to the model structure. The adult phase of the model is a distinct phase in which a different model structure is used. This was done because the costs of care for autistic adults are very different from those for autistic children.

In several of the economic evaluations identified in the cost-effectiveness review (see Appendix 13), adult outcomes were defined in terms of levels of independence, often with three levels defined (e.g. independent, semi-independent and dependent). The definitions attached to each of these levels varied across the different models, but generally adopted one of two approaches. In the first most commonly used approach, levels of independence were defined with reference to special education and in some cases special education needs were also used to predict adult outcomes. The second approach used evidence from the epidemiological and observational literature, which has categorised adult outcomes based on normative evaluations of independence and social outcomes (see Appendix 13 for an overview of these studies).

This approach has important strengths, as it focuses on the support an individual needs and therefore the categorisation systems are useful in terms of reflecting costs. For example, Penner et al. 54 use a classification system reported in Howlin et al. ,15 which is widely adopted within the epidemiological and observational literature. This classification system is useful because it describes a number of important elements of care, with the definitions for each category, including references to residential status and daily needs, both of which are significant drivers of cost in adulthood. In the context of the Penner et al. 54 model, this also allowed them to link the short-term outcomes reported in the effectiveness studies to long-term adult outcomes.

This approach, however, has important limitations, as several studies have noted poor correlations between HRQoL and indicators of independence in autistic populations. 130 Furthermore, a number of studies have been critical of this type of categorisation of adult outcomes, noting that a more nuanced approach is needed to best define ‘standard concepts of what constitutes a “good” social outcome, [as these] may not always be relevant for people with ASD’. 131

The structure adopted in the adult scenario was therefore based on the classification system developed in Howlin et al. ,15 similar to the approach used in Penner et al. 54 However, noting the cited limitations, and in an attempt to mitigate these, independence levels are only used to indicate care needs and costs in adulthood. HRQoL, instead, continues to be based on the Payakachat et al. 127 algorithm used to estimate quality of life in children.

The structure adopted in the adult phase to estimate costs is based on five levels of independence: (1) ‘completely independent’, (2) ‘mostly independent’, (3) ‘some independence’, (4) ‘mostly dependent’ and (5) ‘completely dependent’. The definitions used for each category are based on those used in Howlin et al. 15 and are described in Table 9, and the costs attached to each independence level are described in Resource use.

Independence levels are determined on entering the adult phase of the model based on adaptive behaviour scores at 18.5 years of age. Owing to limitations in the epidemiological and observational evidence base, the adult phase of the model is static (i.e. once individuals enter the adult part of the model, the same level of independence is assumed throughout adulthood). This simplifying assumption is made because of lack of data regarding changes in independence over time. This may or may not reflect real life, and does not include any consideration of an individual’s potential for change or the impact of any additional health or mental health needs, life events or changes in the wider family or social community. This assumption therefore necessarily represents an important limitation of this scenario.

Treatment effect

In the TAU and eclectic arm of the model, changes in cognitive ability and adaptive behaviour scores are modelled using autism-specific natural history data to predict changes over time. To model the treatment effect, in the early intensive ABA-based arm, cognitive ability and adaptive behaviour scores are modelled by applying the treatment effect derived from the IPD meta-analysis. Outcome scores in the early intensive ABA-based interventions arm of the model are therefore the sum of the score predicted from the natural history data plus the treatment effect.

Modelled population

In line with the IPD meta-analysis, the modelled population was assumed to consist of preschool children with a diagnosis of autism. Baseline characteristics modelled included age and sex, as well as the following outcome measures: autism symptom severity (ADOS), cognitive ability (IQ) and adaptive behaviour (VABS). Baseline characteristics were drawn from a pooled analysis of the studies comparing early intensive ABA-based interventions with TAU or eclectic interventions. 84–86,89–94,97,104 The exception to this is age, which was rounded down from 38 months to 36 months to ensure that the time horizon is a constant 15.5 years (this has a negligible impact on the predictions of the model). Note that because the model assumes that children will receive early intensive ABA-based interventions for 2 years, the starting age implies that some children will receive early intensive ABA-based interventions in place of regular schooling. Children in the UK are mandated to attend school by their fifth birthday144 but typically start in the September after their fourth birthday. This assumption was validated by the authors of the effectiveness studies and experts within our Advisory Group, who agreed that at least a proportion of children would receive early intensive ABA-based interventions in a school setting. In a scenario analysis in which effectiveness data were drawn from only UK studies, baseline characteristics were (when possible) drawn from the UK studies only. Starting characteristics used in the base-case analysis and in the UK studies only scenario are summarised in Table 14.

Intervention and comparators

Reflecting the main comparison in the IPD meta-analysis, the model compares early intensive ABA-based interventions with TAU or eclectic interventions.

Health-related quality-of-life data used in cost-effectiveness analysis

As described in Model development, a systematic review was undertaken to identify appropriate utility data with which to populate the model. The studies identified in this review led to the adoption of a statistical algorithm developed in Payakachat et al. 127 to predict HRQoL benefits.

Resource use

Relevant costs and resources were identified in a systematic review of the literature and appropriate routine sources, such as Personal Social Services Research Unit (PSSRU). 146 The studies identified in this review were used to assign appropriate costs associated with the delivery and care of autistic children and adults. Information on the precise description of resources required for each individual therapy was partially based on data derived from the IPD review, and augmented with further information obtained from relevant experts and data sourced from local authorities. The price year of the analysis was 2016/17, as this was the most recent year of publication for PSSRU146 and inflation indices for 2017/18 were not available at the time of writing. Prices reported in alternative cost years were inflated using inflation indices reported in PSSRU. 146

Eligible population

The prevalence of diagnosed autism was estimated to be 3.8 per 1000 for boys and 1.5 per 1000 for girls, with estimates drawn from Taylor et al. 149 Taylor et al. 149 estimated the prevalence of diagnosed autism using data from a large and representative sample of UK general practices. Note that this is lower than estimated prevalence of autism at about 1%,1 as this figure is based on adults and includes undiagnosed cases. The Taylor et al. 149 study estimated prevalence at 8 years of age; therefore, to estimate the proportion of these children who would be diagnosed before the age of 4 years, data on diagnosis patterns were drawn from Brett et al. 150 This was combined with the prevalence rate reported in Taylor et al. 149 by assuming that diagnosis was log-normally distributed, with mean and SD based on data reported in Brett et al. 150 This exercise predicted prevalence of diagnosed autism in children aged < 4 years of 2.5 per 1000 boys and 0.8 per 1000 for girls.

The incident population each month was estimated as one-twelfth of the prevalent population, assuming that children are diagnosed at equal rates throughout the year, with constant prevalence over time.

The total number of children aged < 4 years was drawn from Office for National Statistics mid-2018 population30 estimates, with no population growth assumed for future years.

Scenario analysis

• UK evidence used to estimate short-term intervention effect.

• Time horizon extended to a lifetime time horizon to incorporate adult outcomes.

• Educational placement estimated directly from the effectiveness studies.

• Duration of early intensive ABA-based intervention increased to 36 months.

Sensitivity analysis

• Change in cognitive ability and adaptive behaviour natural history scores per annum.

• Effects of early intensive ABA-based interventions (cognitive ability and adaptive behaviour scores) at 2 years.

• Costs of early intensive ABA-based interventions and TAU and eclectic interventions per annum.

• Costs of education categories, supported placement and specialist schooling.

Deterministic results

Initial results are presented assuming a narrow health-care perspective and therefore only costs and benefits accruing to the health-care system are included. Results of this analysis are presented in Table 20. Full results with a break down of costs are presented in Appendix 19.

In the pessimistic scenario, early intensive ABA-based interventions are associated with £57,879 in additional costs and generates 0.24 additional QALYs. The resulting ICER is £236,837 per additional QALY. In the optimistic scenario, early intensive ABA-based interventions are associated with £57,233 in additional costs and generates 0.84 additional QALYs. The resulting ICER is £68,362 per additional QALY. Using NICE decision rules to benchmark the results of the cost-effectiveness analysis and adopting a £30,000 per QALY threshold, these results indicate that for early intensive ABA-based interventions to be considered cost-effective in the pessimistic scenario, it would be necessary for there to be a further 1.68 QALYs or £50,547 in additional cost savings not captured by the economic model. In the optimistic scenario, early intensive ABA-based interventions would be cost-effective at a threshold of £30,000 per QALY if there were a further 1.07 QALYs or £32,117 in additional cost savings not captured by the model.

A comparison of the incremental costs in the pessimistic and optimistic scenarios shows only a relatively small difference. This reflects the lack of scope for early intensive ABA-based interventions to generate significant cost savings in the health service sector. This similarity in incremental costs, however, contrasts with a comparison of health benefits produced, which are significantly larger in the optimistic scenario.

Building on this initial scenario, Table 21 presents results considering a wider public sector perspective.

In the pessimistic scenario, early intensive ABA-based interventions are associated with £43,940 in additional costs and generates 0.24 additional QALYs. The resulting estimated ICER is £179,799 per additional QALY. In the optimistic scenario, early intensive ABA-based interventions are associated with £36,242 in additional costs and generates 0.84 additional QALYs. The resulting estimated ICER is £43,289 per additional QALY. The adoption of a public sector perspective has a significant impact on incremental costs in both the pessimistic and optimistic scenarios, with incremental costs falling by £13,939 and £20,991, respectively. This is because improvements in cognitive ability and adaptive behaviour could have a large impact in public sector cost savings, which are predominantly caused by changes in education placement.

These differences in incremental costs have a significant impact on the resulting ICER, which fall appreciably in both the pessimistic and optimistic scenarios. However, in both scenarios they remain firmly above the NICE threshold of £20,000–30,000 per QALY. Assuming a threshold of £30,000 per QALY, and making pessimistic assumptions about the long-term treatment effect, it would be necessary for early intensive ABA-based interventions to generate either a further 1.22 QALYs worth of additional health or non-health benefits, or a further £36,608 in additional costs savings. In the optimistic scenario these fall to either a 0.37 QALYs worth of benefits or £11,126 in cost savings.

Probabilistic results

A probabilistic sensitivity analysis was conducted to account for the effects of parameter uncertainty and cost-effectiveness acceptability generated. The base-case results for the probabilistic model are shown in Table 22.

The results of the probabilistic analysis broadly correspond with those of the deterministic analysis and indicate that further health or non-health benefits or cost savings would need to be generated for early intensive ABA-based interventions to be cost-effective at a threshold of £30,000 per QALY. Assuming a public sector perspective, in the pessimistic scenario it would be necessary for early intensive ABA-based interventions to generate either a further 1.29 additional QALYs or produce £38,790 in additional cost savings. In the optimistic scenario, these values fall to 0.47 additional QALYs and £14,066 in additional cost savings.

Assuming an NHS and social services perspective, the probabilistic results indicate that the probability of early intensive ABA-based interventions being cost-effective at a threshold of £30,000 per QALY is < 1% in the pessimistic scenario and 2.6% in the optimistic scenario. The probability that early intensive ABA-based interventions are cost-effective when assuming a public sector perspective is < 1% in the pessimistic scenario and 23% in the optimistic scenario. Taken at face value, these probabilities of cost-effectiveness indicate that there is relatively little chance that early intensive ABA-based therapy is cost-effective when pessimistic assumptions are made about the long-term effectiveness of early intensive ABA-based interventions, and a modest probability in the scenario when optimistic assumptions are made. Care should, however, be taken not to overinterpret these results, given the limitations of the economic model and the scope for further benefits or cost saving to alter this result.

The degree of decision uncertainty is illustrated in Figure 23, which presents the cost-effectiveness acceptability curve taking an NHS and social services perspective and a public sector perspective, respectively. The probability that early intensive ABA-based interventions are cost-effective when taking an NHS and social services perspective remains close to zero up to a threshold of £84,000 per QALY in the pessimistic scenario and up to £25,000 per QALY in the optimistic scenario. When a public sector perspective is adopted, the probability that early intensive ABA-based interventions are cost-effective begins to depart from zero at a threshold of around £30,000 per QALY under the pessimistic scenario and £1000 per QALY in the optimistic scenario.

FIGURE 23.

Cost-effectiveness acceptability curve.

Scenario analysis

Four scenario analyses were undertaken:

1. UK evidence used to estimate short-term intervention effect

2. time horizon extended to a lifetime time horizon to incorporate adult outcomes

3. educational placement estimated directly from the effectiveness studies

4. duration of early intensive ABA-based intervention increased to 36 months.

In each scenario a key model parameter or assumption was varied, with all other parameters fixed at base-case values. Each scenario is presented in turn, along with a short summary of the impact the analysis had on the base-case results. Results of the scenario analysis are summarised in Table 23.

Two-way sensitivity analysis

Two-way and multiway sensitivity analyses were undertaken to explore uncertainty in key input values. This analysis allows assessment of the impact of changing two parameters in the model at the same time. Four analyses were conducted, each considering the following pairs of parameter values:

1. change in cognitive ability and adaptive behaviour natural history scores per annum

2. effects of early intensive ABA-based interventions (cognitive ability and adaptive behaviour scores) at 2 years

3. costs of early intensive ABA-based interventions and TAU or eclectic interventions per annum

4. costs of education categories, supported placement and specialist schooling.

For each analysis, a heat map was generated, depicting the parameter space over which early intensive ABA-based interventions would be considered cost-effective based on NICE thresholds. Cells shaded light blue display ICERs < £20,000 per QALY, orange shading displays ICERs between £20,000 and £30,000 per QALY and purple shading displays ICERs > £30,000 per QALY.

Systematic review and individual participant data meta-analysis

We identified 20 studies, including a total of 894 participants, comparing early intensive ABA-based interventions with alternative interventions (mostly ‘eclectic’ treatment or TAU). Fifteen studies,26,84–86,89–99,103,104 representing 82% of published IPD (n = 654), were received and recoded to a common format, as set out in the project data dictionary, and then reanalysed in accordance with a prespecified analysis plan. One additional study,102 identified in initial searches but not published until after completion of our analyses, was also incorporated into aggregate meta-analyses (see Appendix 12 for details).

All children in the included studies had a diagnosis of autism, ASD or (in some earlier studies) pervasive developmental disorder, and were aged between 24 and 66 months at baseline. The majority of studies recruited participants from the USA, with studies also conducted in the UK, Norway, Australia and Canada. The most commonly reported outcomes were measures of cognitive ability (both verbal and non-verbal) and adaptive behaviour measured using VABS. Studies also less frequently reported measures of autism symptom severity, language comprehension and educational placement.

All of the included studies were at risk of bias on at least one domain using the Cochrane Risk of Bias or ROBINS-I tools,58 with the majority of studies at risk of bias in multiple domains. Randomisation was possible (as shown in a minority of studies) but rarely conducted. Most studies used convenience samples, with allocation to early intensive ABA-based interventions often based on parental preference. There is evidence to suggest that the use of convenience samples may have led to imbalances in participant characteristics between the intervention and comparator groups, with differences in terms of socioeconomic status and the use of co-interventions noted in some studies.

The nature of the interventions meant that blinding of education staff and participants was not possible, but outcome assessors were also often not blinded to intervention. This is particularly problematic when parent-reported measures were used, as the use of convenience samples often meant that parents had actively sought for their child to receive early intensive ABA-based interventions and in some studies parents may also be the source of funding for the intervention. As such, they are likely to have pre-existing expectations about the effectiveness of early intensive ABA-based interventions, which may influence their assessment of parent-reported outcome measures.

It was not possible to assess whether or not these issues and others have had an impact on the results of the included studies or, indeed, the direction of any biases. However, there is a risk that the magnitude of effects observed in our IPD meta-analysis may overestimate the true effects of early intensive ABA-based interventions.

Compared with ‘eclectic’ intervention or TAU, early intensive ABA-based interventions had minimal or no impact on parent-reported adaptive behaviour scores after 1 year (MD 2.93, 95% CI –1.90 to 7.76), but showed an average 7-point difference after 2 years (MD 7.00, 95% CI 1.95 to 12.06). This effect was, however, variable across studies, with treatment effects ranging from a 32-point difference in favour of early intensive ABA-based interventions to a 5-point advantage for TAU and eclectic comparators.

An average improvement in measures of cognitive ability of around 10 points was observed at 1 year (MD 9.16, 95% CI 4.38 to 13.93) and 2 years (MD 14.13, 95% CI 9.16 to 19.10) for early intensive ABA-based interventions relative to TAU and eclectic comparators. Only one study96 provided longer-term follow-up, reporting outcomes at 7 years. In this study no differences between early intensive ABA-based interventions and eclectic treatment were observed on adaptive behaviour (parent-rated VABS) or cognitive ability (IQ) scores.

Results for other outcomes were limited owing to the small numbers of studies reporting them. Autism symptom severity was not measured in most studies and therefore the data available were too limited to be conclusive, with no clear evidence of a difference between early intensive ABA-based interventions and TAU and eclectic groups. There were similarly limited data on language (comprehension and/or expression). Three studies that used the RDLS scale found some evidence that comprehension scores improved by around 12 points when using early intensive ABA-based interventions (MD 11.78, 95% CI 2.12 to 21.45), relative to TAU and eclectic interventions. By contrast, two studies that used the MSEL scale found no difference in scores between early intensive ABA-based arms and TAU or eclectic arms. Only one study90 reported data on behaviours that challenge. The limited data available from this single study showed no clear evidence of any benefit.

A small number of studies reported primary education placement and suggested that children receiving early high-intensity ABA-based intervention were much more likely to be in mainstream education (possibly with support) than children receiving TAU or eclectic interventions or low-intensity ABA-based interventions. However, there is a possibility that this finding was biased by study procedures, such as in Lovaas,26 in which all children receiving early intensive ABA-based interventions were placed in mainstream schools and major efforts were made to avoid authorities subsequently moving them to specialist education placements. Parental choice may also have influenced educational placement, and so these results may also be biased by parental expectations of ABA-based interventions. Additionally, local contextual detail was not available for most studies, so we could not establish the extent to which, for example, inclusive education policies would influence school placement outcomes. This raises questions about the usefulness of school placement as an outcome.

Interpreting the meaningfulness of these observed effects in terms of their impact on the everyday lives of autistic children and parents is not straightforward. Reference to conventions in interpreting effect sizes would imply that the observed effects on both cognitive ability and adaptive behaviour represent small to medium effects. Furthermore, it should be noted that the outcome measures included were considered by the stakeholders in our Advisory Group to be limited in terms of both their ability to reflect benefits and their relevance to practice.

The results of this meta-analysis were consistent with results in previous reviews. For example, Reichow et al. ,45 a recent review published by the Cochrane and Campbell Collaborations, identified similar statistically significant effects on both cognitive ability and adaptive behaviour, and a lack of evidence to support treatment effects on other measures, including behaviours that challenge and autism symptom severity. Similar findings have also been reported in other previous reviews. 4,24,42–44,46–49

A major advantage of conducting IPD meta-analysis is the ability to examine potential moderators of the treatment effect. There were, however, sufficient baseline data to examine only a few factors, namely age, sex, IQ at baseline and VABS composite score at baseline, and, at the study level, classification of intervention type (clinic or community), delivery setting, planned treatment intensity and duration, and year of publication. There was no evidence that any factor moderated any outcome at any time point. This result may reflect the limited data available to detect such effects. However, the magnitude of effects estimates did not suggest that any factor could be a potential moderating factor. Analysis of only UK-based studies suggested smaller treatment effects in these studies. These differences were, however, not statistically significant and could be due to a variety of factors other than location.

Cost-effectiveness

The limitations in the evidence available to the IPD meta-analysis had an important role in shaping the economic model, as a number of potentially relevant outcomes were not captured in the included studies. The structure of the model was therefore designed around the two most commonly measured outcomes, (1) cognitive ability and (2) adaptive behaviour, and so may have missed costs or benefits associated with other outcomes. The economic model adopted a cohort approach to capture changes in these measures and drew on the IPD meta-analysis results to link these outcomes to HRQoL and costs. Furthermore, given the limited evidence about long-term effectiveness, the model explored two scenarios: (1) an optimistic scenario, in which the observed benefits of early intensive ABA-based interventions were assumed to persist; and (2) a pessimistic scenario, in which the proposed benefits dissipate over time.

Given that the implementation of an effective early intensive ABA-based intervention for young autistic children would be likely to have an impact across multiple sectors, the model considered several perspectives relevant to a UK setting, including a health-care and social services perspective and a broader public sector perspective, which included costs falling on the local authority and education sector.

In line with the results of the IPD meta-analysis, the results of the cost-effectiveness model indicated that early intensive ABA-based intervention is more effective (higher incremental QALYs) than TAU or eclectic interventions. The model also indicated that it is more costly. Interpretation of whether or not these additional costs represent value is not straightforward, as the typical decision rules commonly employed in the context of health-care decision-making do not necessarily apply to autism owing to its impact on multiple sectors. These decision rules do, however, provide a useful benchmark against which the results of the cost-effectiveness analysis can be compared. On this basis, and adopting a public sector perspective, the results of the cost-effectiveness analysis indicate that early intensive ABA-based interventions would not meet the decision criteria for implementation used by NICE. This is irrespective of what assumptions are made about the long-term effectiveness of early intensive ABA-based interventions. For early intensive ABA-based interventions to be considered cost-effective according to these criteria, they would need to generate either further benefits or cost savings beyond those captured by the model. In monetary terms, the value of these benefits would need to be £38,790 when pessimistic assumptions were made and £14,066 when optimistic assumptions were made. Care, however, should be taken not to overinterpret the results, as the limitations of the available evidence may mean that it is plausible that uncaptured costs and benefits could alter this result.

The results of the scenario analyses identified a number of drivers of value and showed that early intensive ABA-based interventions would be cost-effective at NICE thresholds only if either significant benefits were accrued in adulthood or the intervention could significantly affect the type of school attended, as was observed in a small number of the effectiveness studies. 17,26,103 The value generated by early intensive ABA-based interventions is, however, contingent on treatment effects persisting for the time horizon of the model and it is much less likely that early intensive ABA-based interventions represent value for money if benefits are realised for only a short period of time.

The magnitude of gains in both cognitive ability and adaptive behaviour scores predicted by the IPD meta-analysis was also a significant driver of cost-effectiveness. This may be of significance when interpreting the results of the economic analysis, given both the significant variations in treatment effects observed across studies included in the IPD meta-analysis and the concerns about bias in these studies. The scenario analysis conducted showed that, conditional on treatment effects persisting, early intensive ABA-based interventions could represent value for money (at thresholds used by NICE) if treatment effect sizes were in line with the more positive studies included in the IPD meta-analysis. Equally, if treatment effects align with the less positive studies or have been exaggerated by confounding and other biases, it is unlikely that early intensive ABA-based interventions represent value for money.

Limitations and uncertainties in the identified evidence

Many of the limitations of this study reflected the limitations of the primary studies identified in the systematic review and included in the IPD meta-analysis. These are discussed in turn below.

Cochrane Central Register of Controlled Trials via Wiley’s The Cochrane Library

Search date: 3 August 2017.

Date range searched: inception to 3 August 2017.

Records retrieved: 193.

Search name: ABA autism.

Last saved: 3 August 2017.

Cumulative Index to Nursing and Allied Health Literature via EBSCOhost

Search date: 8 August 2017.

Date range searched: inception to 8 August 2017.

Records retrieved: 1530.

Conference Papers Citation Index via Web of Science

Search date: 8 August 2017.

Date range searched: inception to 8 August 2017.

Records retrieved: 46.

EMBASE via OVID

Search date: 3 August 2017.

Date range searched: inception to 3 August 2017.

Date range searched: 1974 to 2017 week 31.

Records retrieved: 2127.

Education Resources Information Center via EBSCOhost

Search date: 8 August 2017.

Date range searched: inception to 8 August 2017.

Records retrieved: 2070.

Electronic Theses Online Service (The British Library)

Search date: 2 August 2017.

Date range searched: inception to 2 August 2017.

URL: http://ethos.bl.uk/Home.do

MEDLINE via OVID

Search date: 3 August 2017.

Date range searched: inception to 3 August 2017.

Records retrieved: 1626 (after deduplication).

PsycINFO via OVID

Search date: 3 August 2017.

Date range searched: 1806 to July week 5 2017.

Records retrieved: 2248.

Social Science Citation Index via Web of Science

Search date: 8 August 2017.

Date range searched: inception to 8 August 2017.

Records retrieved: 978.

ClinicalTrials.gov

Via: https://clinicaltrials.gov/ct2/home

Search date: 8 August 2017.

Date range searched: inception to 8 August 2017.

Ran a series of searches, downloaded records and deduplicated within EndNote bibliographic software.

World Health Organization International Clinical Trials Registry Platform

Via: http://apps.who.int/trialsearch/Default.aspx.

Search date: 8 August 2017.

Date range searched: inception to 8 August 2017.

Ran a series of searches, downloaded records and deduplicated within EndNote bibliographic software.

Illustration A: study-level details

Illustration B: participant, family and treatment characteristics at baseline

Illustration C: key to collected outcome measures

Illustration D: outcome measure values at baseline and follow-up

Whenever possible for continuous data, please provide exact rather than categorical values [e.g. ‘64’ rather than ‘≤70’ for WISC, Fourth Edition, full-scale IQ].

Please include the full name and version of each outcome measure that was used.

Participants

Studies that include children with a diagnosis of autistic disorder, Asperger syndrome, PDD-NOS, atypical ASD or ASD.

Interventions

Intensive behavioural interventions based on ABA, defined as:

• > 15 hours per week

• comprehensive, targeting a range of behaviours not a single behaviour (e.g. joint attention)

• ABA-based teaching strategies by qualified and trained staff

• delivered at least initially on a one-to-one basis

• qualified supervision of the therapist delivering the intervention.

Studies that involve a degree of parental involvement will be included if the other criteria are satisfied.

Comparators

Relevant comparators will include all other forms of early intervention.

Outcomes

Inclusion will not be restricted by outcome.

Study designs

Prospective RCTs and non-RCTs meeting all other inclusion criteria will be eligible for inclusion.