Download Three Dimensions of Oppositionality in Autism

J Abnorm Child Psychol
DOI 10.1007/s10802-013-9778-0
Three Dimensions of Oppositionality in Autism
Spectrum Disorder
William Mandy & Laura Roughan & David Skuse
# Springer Science+Business Media New York 2013
Abstract In autism spectrum disorder (ASD), symptoms of
oppositional defiant disorder (ODD) are common but poorly
understood. DSM-5 has adopted a tripartite model of ODD,
parsing its features into ‘angry and irritable symptoms’ (AIS),
‘argumentative and defiant behavior’ (ADB) and ‘vindictiveness’. This was based on findings in non-autistic populations
that each of these dimensions of oppositionality has a distinct constellation of associations with internalising and
externalising psychopathology. We applied the tripartite
DSM-5 ODD model to ASD to test its generalisability beyond
non-ASD populations; and to elucidate the nature of ODD
symptoms in ASD. Participants were 216 verbally-fluent
young people (mean age=9.6 years, range 3.0 to 16.2 years,
82 % male) with ASD. Cross-sectional parent-and teacherreport data were analysed using bootstrap multiple regression
to test the following predictions, derived from studies of nonASD young people: (1) AIS will be the main predictor of
internalising problems; (2) ADB will be the main predictor of
ADHD symptoms; (3) all ODD traits will independently predict conduct disorder symptoms; (4) vindictiveness will be
the main predictor of aggressive conduct problems. Our findings using both parent and teacher data were consistent with the
non-ASD ODD literature. AIS were associated with internalising but not externalising problems; ADB and vindictiveness were associated with externalising but not internalising
problems; and vindictiveness was the main predictor of aggression. The DSM-5 tripartite model of ODD appears to be
W. Mandy (*)
Research Department of Clinical, Educational and Health
Psychology, UCL, Gower Street, WC1N 6BT London, UK
e-mail: [email protected]
L. Roughan
Department of Child and Adolescent Mental Health, Great Ormond
Street Hospital for Children, London, UK
D. Skuse
Behavioural and Brain Science Unit, UCL Institute of Child Health,
Guilford Street, London, UK
generalisable to ASD: for people with an autistic disorder, AIS,
ADB and vindictive dimensions of oppositionality have distinct
associations with concurrent psychopathology, suggesting the
need to assess them as separate constructs.
Keywords Autism spectrum disorder (ASD) . Oppositional
defiant disorder (ODD) symptoms . Conduct disorder
symptoms . Angry and irritable symptoms . Argumentative
and defiant behavior . Vindictiveness . Diagnostic and
Statistical Manual—Fifth Edition (DSM-5)
Whilst the core and definitive features of autism spectrum
disorder (ASD) are deficits in social reciprocity, communication and flexibility, most people with ASD experience a constellation of additional difficulties, including emotional and
behavioural disturbance (Gadow et al. 2008b). This is reflected
in the frequency with which non-autistic psychiatric diagnoses
are applicable to people with ASD. The only populationrepresentative study to investigate psychiatric comorbidity in
ASD found that 70 % of their sample of 12-year-olds met
criteria for at least one additional mental disorder, and 41 %
met criteria for two (Simonoff et al. 2008). This is consistent
with studies of clinically-ascertained samples which have applied DSM-IV diagnoses and found similarly elevated psychiatric comorbidity (e.g. Gadow et al. 2004; Gadow et al. 2005).
High rates of co-occurring psychiatric symptomatology persist
into adolescence (Simonoff et al. 2013), and have also been
observed in adult ASD samples (Lugnegård et al. 2011).
In addition to its high prevalence, a striking feature of the
non-autistic psychopathology that accompanies ASD is its
variety. Simonoff and colleagues (2008) discovered an array
of internalising and externalising problems in their community
ASD sample. The most common diagnoses were of anxiety
disorders (48 % had at least one), followed by attention deficit
hyperactivity disorder (ADHD; 28 %) and oppositional defiant
disorder (ODD; 28 %). A similar picture has emerged from the
study of clinically ascertained ASD samples, in which high
rates of internalising difficulties (van Steensel et al. 2011),
ODD and ADHD (Gadow et al. 2005) have been found.
J Abnorm Child Psychol
Relatively little is known about the processes that underpin the development and maintenance of behavioural and
emotional disturbances in people with ASD, and this limits
quality of clinical care. One approach to increasing knowledge in this area, which we take in the current study, is to test
whether discoveries about the development of psychopathology in non-autistic populations apply to people with ASD.
We argue that such an approach offers the promise of ‘quick
wins’ in improving clinical practice, by identifying insights
and clinical technologies based on non-autistic populations
that could usefully be applied to ASD. Furthermore, testing
the applicability to ASD of general models of psychopathology can also highlight areas of divergence, where superficially similar behavioural or emotional difficulties turn out to
have distinct causes and treatment needs in ASD and nonASD groups (Gadow et al. 2004; Simonoff et al. 2013).
In the current study we test whether recent advances in
understanding ODD, achieved through the study of nonautistic people, can be usefully applied to ASD. We have
focused on ODD as it is one of the most commonly diagnosed
comorbidities in ASD (Simonoff et al. 2008), and because it is
a key construct within developmental psychopathology, operating as a risk factor for a wide variety of internalising and
externalising difficulties (Stringaris and Goodman 2009a). As
such, increasing understanding of ODD in ASD may offer
insights into mechanisms underpinning the broad range of
psychopathology found in autistic populations.
ODD is characterised by ‘a pattern of negativistic, defiant,
disobedient or hostile behaviour towards authority figures’
(American Psychiatric Association 2000, p.100), and represents the extreme of a trait distribution that extends throughout the population (Sterba et al. 2010). ODD and ODD
traits (‘oppositionality’) are a risk factor for a spectrum of
internalising and externalising problems, and cannot simply
be understood as early and/or mild manifestations of disruptive behaviour. This multifinality, and some factor analytic
work, has led to the hypothesis that oppositionality is a multidimensional construct that should be parsed into separate
factors, each with partially distinct associations with different
types of maladaption (Krieger et al. 2013; Stringaris and
Goodman 2009a; 2009b).
The DSM-5 separates the features of ODD into ‘angry and
irritable symptoms’ (AIS), ‘argumentative and defiant behaviour’ (ADB) and ‘vindictiveness’ (American Psychiatric
Association 2013).1 This partitioning of ODD symptomatology is based on studies of non-autistic community and clinical
samples that suggest each of these symptom dimensions has
The DSM-5 constructs of ‘angry/irritable symptoms’, ‘argumentative disruptive behavior’ and ‘vindictiveness’ are sometimes referred to as ‘irritable’, ‘headstrong’ and ‘hurtful’ dimensions of oppositionality, notably by
Stringaris and Goodman (2009a; 2009b). In the interests of clarity, we have
chosen to use only the DSM-5 nomenclature when reviewing the literature.
1
distinct associations with other forms of psychopathology.
AIS, but not ADB and vindictiveness, independently predict
concurrent (Stringaris and Goodman 2009a) and future
(Leibenluft 2011; Rowe et al. 2010; Stringaris and Goodman
2009b) mood and anxiety disorders. Of the three ODD dimensions, ADB behaviour is the best predictor of ADHD
(Stringaris and Goodman 2009a; 2009b). All three factors
are associated with conduct disorder symptoms, although
cross-sectional and longitudinal associations between AIS and
conduct problems have been observed to be weak (Stringaris
et al. 2012), or non-existent when controlling for other ODD
traits (Stringaris and Goodman 2009b). Vindictiveness is especially associated with aggressive symptoms of CD and
callous-unemotional behaviour (Stringaris and Goodman
2009a; 2009b).
We investigated whether this specific pattern of associations
holds in an ASD sample, in order to test the validity of a three
factor model of ODD traits in ASD. Specifically we tested the
following hypotheses in a sample of children and young people
with ASD: (1) AIS, but not ADB or vindictiveness, will be
uniquely associated with internalising problems; (2) ADB will
be particularly associated with ADHD symptoms; (3) AIS,
ADB and vindictiveness will all be associated with CD symptoms; (4) Vindictiveness will be particularly associated with
aggressive CD symptoms. Some phenomena of ODD in ASD
have been found to be informant-specific, meaning that they
vary depending on whether a parent or teacher provided data
(Gadow et al. 2008a). On this basis we tested our hypotheses
using information collected from both parents and teachers.
We argue that our test of a tripartite model of ODD could
offer insights into the variety of emotional and behavioural
difficulties that co-occur with ASD. Furthermore, by testing
whether findings on ODD from non-autistic samples are
relevant to people with ASD, we can contribute to debates
about whether ODD behaviours in ASD are manifestations
of true comorbidity, or if instead they are a phenocopy,
representing an autistic epiphenomenon, with only superficial resemblance to true ODD symptoms.
Methods
Participants
Participants were 216 children and adolescents with ASD
(82 % male) who received a full ASD and comorbidity assessment at a specialist social communication disorders service in
London, UK. Information about their age, IQ, ASD symptoms
and comorbidities are presented in Table 1. IQ data were only
available for a subsection of the sample (verbal IQ available for
72 %, performance IQ for 56 %). This was not because a
decision whether or not to test IQ was dependent on a child’s
particular presenting problems; it occurred because IQ testing
J Abnorm Child Psychol
Table 1 Characteristics of the sample (n=216)
Age, IQ and autistic symptomatology
Mean (SD) [Range]
Age in years
9.1 (2.9) [3.1 to 16.2]
3Di Reciprocal Social Interaction
15.1 (3.4) [9.0 to 24.2]
3Di Communication
13.9 (3.4) [6.3 to 22.2]
3Di Repetitive
4.3 (2.4) [0.0 to 11.5]
Verbal IQ1
93.6 (19.9) [40 to 153]
Performance IQ2
94.5 [57 to 143]
Comorbidity
Proportion in SDQ
‘definite problems’ range
SDQ Total problems
77.8 %
SDQ Hyperactivity
68.1 %
SDQ Conduct problems
56.5 %
SDQ Emotional problems
49.1 %
SDQ Peer problems
80.1 %
3Di developmental, dimensional and diagnostic interview; SDQ strengths
and difficulties questionnaire; 1 Verbal IQ n=156; 2 Performance IQ n=122
only became a routine part of the clinic’s assessment process
after the start of the current study’s sampling period. There
were no significant differences in level of ASD, CD or ODD
symptomatology between participants with and without IQ
data. Of the sample 81 (37.5 %) received a diagnosis of
DSM-IV autistic disorder, 62 (28.7 %) were diagnosed with
Asperger’s disorder and 73 (33.8 %) met criteria for pervasive
developmental disorder—not otherwise specified. This is a
predominantly high-functioning sample, with all participants
demonstrating fluent language and attending mainstream
school at the time of assessment, as these are referral criteria
of the clinic in which this study was based. Nevertheless,
autistic symptomatology and comorbidity was substantial and
severe, equivalent to that found in a community-ascertained
UK sample (Baird et al. 2006; Simonoff et al. 2013).
Procedure
All young people were assessed using the Dimensional,
Developmental and Diagnostic Interview (3Di; Skuse et al.
2004), administered by an experienced child and adolescent
psychiatrist or clinical psychologist. The 3Di is a parent-report
interview that uses computerised algorithms to generate severity scores for each element of the autism triad, and also
collects information from parents and teachers on symptoms
of ODD and CD. In addition, the Strengths and Difficulties
Questionnaire (Goodman 1997) was administered to parents
and teachers. ASD diagnosis was based on information from
the 3Di, plus structured reports from the child’s school or
nursery and direct observation of the child in clinic, which
was performed using the Autism Diagnostic Observation
Schedule (ADOS; Lord et al. 2000) in most (n=167, 77 %)
cases. This retrospective chart review study used nonidentifiable data collected in the course of routine clinical
assessment. Ethical permission was provided after review by
the research and development office of the hospital in which
this study was based, and appropriate measures were taken to
ensure young person and parent confidentiality.
Measures
The Developmental, Dimensional and Diagnostic Interview
(3Di)—The 3Di is a validated parent report interview designed
for the assessment of ASD and related psychopathology (Skuse
et al. 2004). It covers developmental history and a range of past
and current symptoms. A subset of 122 questions are used in an
ASD diagnostic algorithm that generates dimensional scores
for social reciprocity, social communication and repetitive,
stereotyped behaviour (RSB). The test-retest and interrater
reliability of the 3Di’s ASD algorithm is strong, with all
intraclass correlations in the psychometric evaluation study
exceeding 0.85 (Skuse et al. 2004). In terms of criterion validity, the 3Di shows excellent agreement with clinician ASD
diagnosis (sensitivity=1; specificity=0.97). Also there is a high
level of agreement between the 3Di and Autism Diagnostic
Interview-Revised algorithms on whether a case is above the
clinical threshold for social reciprocity (86 %), communication
(97 %) and RSB (76 %) (Skuse et al. 2004).
In addition to its ASD algorithm, the 3Di measures DSM-IVdefined symptoms of ODD and CD, by both parent and teacher
report, with each relevant item directly corresponding to a
DSM-IV diagnostic criterion. Inter-rater and test-retest reliability for diagnoses based on these sections of the 3Di is excellent,
with greater than 95 % agreement between and within clinicians
in the original validation study. Criterion validity, assessed by
comparing 3Di and clinician diagnoses, is also excellent for
both ODD (91 % agreement) and CD (93 % agreement).
In the 3Di, parents report on the presence of each symptom of
ODD, with answers coded as ‘not present’, ‘possibly present’,
or ‘definitely present’. The three parent-report ODD subscores
were calculated on the basis of guidelines proposed for DSM-5.
To generate an AIS score, three items measuring frequent
temper loss, touchiness and anger/resentfulness were summed
(Cronbach’s α=0.72 in the current sample). ADB scores were
summed from the four items measuring arguing with adults,
refusal to comply, deliberately annoying others and blaming
others for mistakes (α=0.77). The vindictiveness score
was based on response to the 3Di item measuring spiteful or vindictive behaviour. Teacher ODD scores were
calculated in the same way, with one exception. Teachers
were not asked about the ‘touchiness’ symptom of ODD, so
the teacher-report AIS score was calculated from two items
concerning temper tantrums and anger/resentfulness (Kendall’s
ΤB =0.66, p<0.001). The teacher ADB scale showed good
reliability (α=0.86).
J Abnorm Child Psychol
Conduct disorder total scores were summed from 3Di items
measuring DSM-IV defined symptoms. By both parent and
teacher report, each symptom was marked as present or absent
within the previous 12 months. We summed these items to
derive total CD symptom scores from parent (α=0.79) and
teacher (α=0.85) data. Also, we used all CD items to derive
distinct subscales for aggressive and rule-breaking CD symptoms. Based on previous factor analytic work (Tackett et al.
2005), and DSM-IV, the aggressive CD scale by parent report
was summed from items measuring the following symptoms
of CD: bullying, fighting, use of weapons, cruelty to people,
cruelty to animals, sexual assault and stealing while confronting a victim (α=0.60 in parent data). The rule-breaking CD
scale from parent report was summed from items measuring
fire setting, other destruction of property, stealing without
confrontation, breaking into property, often staying out beyond curfew, running away, truanting and lying (α=0.71).
The teacher-rated conduct scales were identical in composition with one exception: the rule-breaking scale did not include items measuring breaking into property, staying out
beyond curfew and running away from home. Teachers were
not asked to rate the presence of these symptoms, as it was
thought that they would not have the requisite knowledge of
the child’s behaviour to provide a valid answer. Internal consistency was acceptable for both teacher-report aggressive
(α=0.79) and rule-breaking (α=0.67) CD scales.
The Strengths and Difficulties Questionnaire (SDQ)—The
SDQ comprises 25 items in five subscales: conduct problems,
emotional problems, hyperactivity, peer problems and prosocial behaviour (Goodman 1997). Rated by parents and by
teachers, it has acceptable reliability and validity when
assessing adaptation and psychopathology in children and
adolescents both as a screening instrument (Goodman 2001)
and as a dimensional measure for quantifying the degree of an
individual’s strengths and difficulties (Goodman and Goodman
2009). The SDQ appears to offer a valid measure of psychopathology in autistic populations (Simonoff et al. 2013). In the
current study published thresholds for ‘definite’ caseness were
used to derive the comorbidity estimates presented in Table 1
(Goodman 1997). In addition, SDQ ‘emotional problems’ and
‘hyperactivity’ scales were used as dimensional measures of
internalising difficulties and ADHD symptoms respectively.
The SDQ emotional problems scale comprises items measuring anxiety, low mood and somatising. It has been shown to
correlate strongly with the Child Behaviour Checklist’s
internalising scale (r=0.74; Goodman and Scott 1999). The
SDQ hyperactivity scale measures ADHD symptomatology,
comprising items on inattention, hyperactivity and impulsivity.
It correlates strongly with the Child Behaviour Checklist scale
measuring inattention/hyperactivity (r=0.71) and with clinician ADHD diagnosis (area under the curve=0.86; Goodman
and Scott 1999).
Analyses
STATA version 12 (StataCorp 2011) was used. The study
hypotheses were tested using hierarchical multiple regression.
Some of our outcome variables deviated from normality so we
used bootstrap regression based on 1,000 random samples to
ensure robustness of our parameter estimates. This allowed
empirically-based 95 % confidence intervals and p values to
be derived, avoiding any normality-based assumption associated with the t-distribution used in standard linear regression.
Each regression model had a first block containing age, sex,
3Di reciprocal social interaction, 3Di communication and 3Di
repetitive, stereotyped behaviour. These were entered as control variables. The second block comprised AIS, ADB and
vindictiveness scores. ADHD symptoms are associated with
ODD and CD (Burke et al. 2002). Therefore we added the
SDQ hyperactivity subscale as an additional control variable
to parent and teacher models regressed on total CD, aggressive CD and rule-breaking CD symptoms. When more than
one of the ODD dimensions was significant in the final model,
we used the STATA ‘lincom’ command to execute a z-test,
which tested the null hypothesis that the difference between
their coefficients was not different from zero. This enabled us
to test if one dimension of oppositionality made a greater
contribution to the model than another.
Results
As Table 2 shows, in this sample, symptoms of ODD were
common. Eighty-seven percent (n=187) of the sample had at
least one definitely present ODD symptom by parent report,
Table 2 Oppositional defiant disorder symptoms by parent and teacher
report
Parent report
Number with
symptom definitely
present (%)
N=216
Teacher report
Number with
symptom definitely
present (%)
Angry and irritable symptoms
Temper loss
139 (64 %)
78 (36 %)
Angry and resentful
140 (65 %)
75 (35 %)
Touchiness
113 (52 %)
–
Argumentative and defiant behaviour
Argumentative
112 (52 %)
77 (36 %)
Refusal to do as told
113 (52 %)
87 (40 %)
Blames others for mistakes
121 (56 %)
65 (30 %)
Deliberately annoys others
75 (35 %)
60 (28 %)
49 (23 %)
40 (19 %)
Vindictiveness
Spiteful or vindictive
J Abnorm Child Psychol
and 55 % (n=119) had at least four. By teacher report, 56 %
(n=122) participants had at least one definite ODD symptom,
and 31 % (n=66) had four or more. Forty-seven percent
(n=102) of participants had at least one definite CD symptom
by parent report, and 19 % had three or more. The most
common parent-reported CD symptoms were lying (27 %)
and fighting (25 %). Sexual assault, burglary and robbery
involving confrontation were rarest, each reported by parents
in one percent or less of the sample. By teacher report, 32 % of
children had at least one symptom of conduct disorder, and
13 % had three or more. Most commonly observed by teachers
were lying (16 %) and fighting (13 %). Sexual assault, animal
cruelty and fire starting were rare, reported by teachers in one
percent or less of participants.
Correlations between the three dimensions of oppositionality
and the other study variables are presented for parent and
teacher data in Table 3. Associations between ODD traits and
3Di scores were small (<±0.14) and, with one exception,
non-significant. In the parent data AIS correlated with ADB
(r(214)=0.70, p<0.001) and vindictiveness (rs(214)=0.44,
p<0.001). ADB and vindictiveness were also associated
(rs(214)=0.58, p<0.001). By teacher report AIS was associated with ADB (r(214)=0.73, p<0.001) and vindictiveness
(rs(213)=0.57, p<0.001). There was also a strong correlation
between ADB and vindictiveness (rs(215)=0.66, p<0.001).
Tables 4 and 5 show standardised coefficients (β’s) and
their 95 % confidence intervals from the bootstrap hierarchical
regressions for parent and teacher data respectively. Each
regression controlled for age, sex, and 3Di ASD algorithm
scores by including these variables in block 1. The regressions
on total CD, aggressive CD and rule-breaking CD symptoms
also included the SDQ hyperactivity scale in block 1, to control
for ADHD symptoms. For all regressions, the second
block comprised the three ODD dimensions, allowing us
to assess their individual, independent contributions to each
model, above and beyond the control variables.
Hypothesis one—AIS, but not ADB and Vindictiveness,
Will Predict Internalising Problems
Parent and teacher models offered support for hypothesis
one, as in each model AIS was the only ODD dimension
that independently predicted internalising problems, as measured by the SDQ emotional problems subscale. In the parent
data, female sex and repetitive stereotyped behaviour were
also independently associated with internalising.
Hypothesis two—ADB Will be Particularly Associated
with ADHD Symptoms
In the parent-report model both AIS and vindictive behaviour were independently associated with ADHD symptoms,
as measured by the SDQ hyperactivity subscale. Although β
was larger for ADB than for vindictiveness, a z-test revealed
that this difference was not significant (p=0.292). By teacher
report ADB was the only ODD dimension that predicted
ADHD symptoms.
Hypothesis Three—AIS, ADB and Vindictive Dimensions
Will all be Associated with Overall CD Symptoms
The parent model shows that ADB and vindictiveness were
each independently associated with the overall CD symptom
score. Vindictiveness was a significantly stronger predictor
of CD symptoms than ADB (p=0.044). In the teacher model
it was also the case that ADB and vindictiveness, but not
AIS, independently predicted CD symptoms. Whilst the coefficient for vindictiveness was bigger than that for ADB in
the teacher model, our z-test showed that this difference did
not reach significance (p=0.125).
Hypothesis Four—Vindictiveness Will be Particularly
Associated with Aggressive CD Symptoms
The parent and teacher models predicting aggressive symptoms of conduct disorder support hypothesis four. In the
parent and teacher models vindictiveness was the only significant predictor of aggression, and this was a substantial
effect (βs>0.45). We also ran models predicting the rulebreaking symptoms of CD using both parent and teacher
data. They both showed ADB and vindictiveness to be
independently associated with rule-breaking CD symptoms.
In parent (p=0.421) and teacher (p=0.301) models, coefficients for these dimensions of oppositionality were not significantly different in size.
Discussion
In a sample of young people with ASD, we investigated the
DSM-5 model of oppositional defiant disorder (ODD),
which parses features of oppositionality into ‘angry and
irritable symptoms’ (AIS), ‘argumentative and defiant behaviour’ (ADB) and ‘vindictiveness’. To assess the validity
of this division we tested whether these three dimensions of
oppositionality had distinct associations with other measures
of concurrent psychopathology. Specifically we tested the
following hypotheses, which were based on findings from
non-autistic young people: (1) AIS, but not ADB or vindictiveness, will be associated with internalising problems; (2)
ADB traits will be particularly associated with ADHD symptoms; (3) AIS, ADB and vindictive traits will all be associated
with conduct disorder (CD) symptoms; (4) Vindictiveness will
be particularly associated with aggressive CD symptoms.
0.07
–0.09
0.04
–0.06
0.54***
0.46***
0.42***
0.36***
0.37***
Parent rule-breaking CD symptoms
Age
0.54***
0.57***
0.59***
0.14*
Teacher rule-breaking CD symptoms
Teacher SDQ hyperactivity (ADHD
symptoms)
Teacher SDQ emotional problems
(internalising)
Teacher total conduct disorder
symptoms
Teacher aggressive CD symptoms
3Di Repetitive stereotyped behaviour
–0.10
0.32***
3Di Communication
3Di Social
Sex (male=0, female=1)
–0.10
0.01
0.02
0.14*
Vindictiveness1
Teacher report
0.49***
0.39***
0.44***
0.28***
0.36***
0.05
0.60***
0.39***
0.54***
0.21**
0.37***
0.06
–0.07
–0.11
–0.03
–0.02
0.13
–0.16*
–0.10
Argumentative disruptive
behaviour
–0.00
Angry irritable
symptoms
0.57***
0.62***
0.62***
0.22**
0.26***
0.10
–0.13*
–0.12
–0.14*
0.07
Vindictiveness1
0.03 [−0.12 to 0.18]
0.13* [0.00 to 0.27]
–
0.11 [−0.03 to 0.25]
0.08 [−0.06 to 0.23]
–0.02 [−0.13 to 0.10]
–
3Di social Reciprocity
3Di communication
3Di repetitive Stereotyped Behaviour
SDQ hyperactivity (ADHD symptoms)
0.22***
Overall R2 of model
0.20***
0.10***
0.04 [−0.11 to 0.19]
-0.08 [−0.27 to 0.12]
0.37*** [0.18 to 0.56]
0.44***
0.31***
0.45*** [0.31 to 0.59]
0.23** [0.09 to 0.36]
0.04 [−0.08 to 0.16]
0.06 [−0.03 to 0.14]
–0.02 [−0.15 to 0.11]
0.00 [−0.11 to 0.11]
0.05 [−0.06 to 0.16]
–0.01 [−0.12 to 0.09]
0.03 [−0.06 to 0.11]
Total conduct disorder symptoms
0.40***
0.29***
0.47*** [0.32 to 0.63]
0.14 [−0.02 to 0.30]
0.07 [−0.06 to 0.20]
0.07 [−0.03 to 0.17]
–0.00 [−0.13 to 0.12]
0.00 [−0.13 to 0.13]
0.02 [−0.11 to 0.14]
–0.04 [−0.15 to 0.08]
–0.04 [−0.13 to 0.06]
Aggressive conduct
disorder symptoms
0.34***
0.28***
0.35*** [0.21 to 0.49]
0.26*** [0.13 to 0.39]
0.01 [−0.10 to 0.11]
0.03 [−0.05 to 0.12]
–0.03 [−0.18 to 0.12]
0.00 [−0.12 to 0.13]
0.08 [−0.04 to 0.19]
0.01 [−0.11 to 0.13]
0.08 [−0.02 to 0.17]
Rule-breaking conduct
disorder symptoms
SDQ strengths and difficulties questionnaire; 3Di developmental, dimensional and diagnostic interview; ADHD attention deficit hyperactivity disorder; ODD oppositional defiant disorder
*p<0.05, **p<0.01, ***p<0.001
0.15* [0.03 to 0.27]
0.19***
Δ R2 from addition of ODD traits
0.28** [0.11 to 0.44]
Argumentative defiant behaviour
Vindictiveness
0.11 [−0.08 to 0.29]
Angry irritable mood
Block two – ODD variables
0.20** [0.07 to 0.32]
–0.07 [−0.20 to 0.07]
0.05 [−0.09 to 0.19]
0.07 [−0.06 to 0.20]
–0.12 [−0.25 to 0.01]
Sex (male=0, female-1)
SDQ emotional problems (Internalising)
Age
Block one—control variables
SDQ hyperactivity
(ADHD symptoms)
Table 4 Bootstrap regression models testing the study hypotheses in parent-report data, showing standardised betas and their 95 % confidence intervals
3Di developmental, dimensional and diagnostic interview; SDQ strengths and difficulties questionnaire; ADHD attention deficit hyperactivity disorder; 1 All coefficients for Vindictiveness are Spearman’s ρ
*p<0.05, **p<0.01, ***p<0.001
0.24***
0.36***
0.49***
0.40***
0.32***
Parent SDQ hyperactivity (ADHD
symptoms)
Parent SDQ emotional problems
(internalising)
Parent total conduct disorder
symptoms
Parent aggressive CD symptoms
0.04
0.02
3Di Repetitive stereotyped behaviour
3Di Communication
3Di Social
0.24***
0.02
0.27***
0.02
Argumentative disruptive
behaviour
Sex (male=0, female=1)
Angry irritable
symptoms
Age
Parent report
Table 3 Correlations between traits of oppositional defiant disorder and study variables, in parent and teacher data
J Abnorm Child Psychol
SDQ strengths and difficulties questionnaire; 3Di developmental, dimensional and diagnostic interview; ADHD attention deficit hyperactivity disorder; ODD oppositional defiant disorder
*p<0.05, **p<0.01, ***p<0.001
0.35***
0.43***
13***
0.08***
0.45***
0.26***
0.37***
0.26** [0.08 to 0.43]
Overall R2 of model
0.26***
0.44*** [0.24 to 0.64]
0.13***
0.35***
0.09 [−0.10 to 0.27]
−0.03 [−0.20 to 0.14]
Vindictiveness
Δ R2 from addition of ODD traits
0.44*** [0.26 to 0.61]
0.01 [−0.16 to 0.18]
0.41*** [0.20 to 0.62]
0.21* [0.04 to 0.39]
−0.05 [−0.27 to 0.17]
0.27** [0.08 to 0.46]
Argumentative defiant behaviour
0.01 [−0.15 to 0.17]
−0.01 [−0.15 to 0.13]
0.26** [0.06 to 0.45]
0.15 [−0.04 to 0.34]
Angry irritable mood
Block two—ODD variables
0.05 [−0.11 to 0.21]
0.01 [−0.11 to 0.13]
−0.02 [−0.09 to 0.12]
0.06 [−0.05 to 0.18]
0.04 [−0.07 to 0.14]
0.04 [−0.09 to 0.17]
0.04 [−0.11 to 0.18]
−
−0.03 [−0.16 to 0.10]
−
3Di repetitive Stereotyped Behaviour
SDQ hyperactivity (ADHD symptoms)
0.08 [−0.05 to 0.21]
0.03 [−0.11 to 0.16]
0.06 [−0.07 to 0.19]
0.05 [−0.09 to 0.20]
0.09 [−0.06 to 0.25]
0.05 [−0.08 to 0.18]
−0.06 [−0.20 to 0.08]
0.03 [−0.11 to 0.18]
−0.07 [−0.23 to 0.08]
0.02 [−0.12 to 0.17]
3Di social Reciprocity
3Di communication
−0.14 [−0.32 to 0.04]
0.03 [−0.09 to 0.14]
0.06 [−0.04 to 0.16]
0.03 [−0.06 to 0.12]
0.03 [−0.06 to 0.13]
0.00 [−0.10 to 0.10]
Block one—control variables
0.04 [−0.04 to 0.12]
0.21** [0.08 to 0.35]
0.07 [−0.06 to 0.20]
–0.14* [−0.27 to −0.01]
−0.30*** [−0.43 to −0.15]
Age
Sex (male=0, female-1)
Aggressive conduct
disorder symptoms
Total conduct
disorder symptoms
SDQ emotional problems
(Internalising)
SDQ hyperactivity
(ADHD symptoms)
Table 5 Bootstrap regression models testing the study hypotheses in teacher-report data, showing standardised betas and their 95 % confidence intervals
Rule-breaking conduct
disorder symptoms
J Abnorm Child Psychol
Our findings were mostly supportive of the study hypotheses, thus providing evidence for the validity of the DSM-5
tripartite ODD model in ASD. As in studies of non-ASD
samples (Stringaris and Goodman 2009a), we found that AIS
was the only ODD dimension uniquely associated with
internalising problems. This was the case in both parentand teacher-report data. There was support for the second
hypothesis, that ADB would have the strongest association
with ADHD, as ADB was the only variable associated with
ADHD symptoms in both parent and teacher models. In line
with hypotheses three and four, ADB and vindictiveness
were both associated with overall CD symptoms; and by
both parent and teacher report, vindictiveness was the sole
predictor of aggressive CD behaviours. Nevertheless, in
partial contradiction of our third hypothesis, there was no
relationship between AIS and CD symptoms. This contrasts
with cross-sectional (but not longitudinal) analyses in nonASD samples, which have found a modest but significant
relationship between AIS and CD (Stringaris and Goodman
2009a; 2009b). As such, our findings with respect to AIS and
CD symptoms may signify an area of divergence between
ASD and non-ASD populations.
Theorists of comorbidity have debated the meaning of the
high rates of diagnosable additional mental disorders in ASD
(e.g. Gadow et al. 2004; Lecavalier et al. 2009). Is this true
comorbidity, in which the psychiatric symptoms seen in ASD
are genuine manifestations of DSM-IV mental disorders? Or
does it actually reflect the presence of phenocopies which are
epiphenomena of ASD? Our findings speak to this debate,
offering support for the idea that the high rates of diagnosable
ODD in ASD may reflect authentic comorbidity. There were
many more similarities than differences between our findings
and those reported for non-autistic samples (e.g., Stringaris and
Goodman 2009a; 2009b), suggesting that the three dimensions
of ODD configure similarly in autistic and non-autistic populations, both in their relationships with each other and in their
associations with external validators. This is compatible with
the possibility that they share a similar underlying structure and
aetiology. Furthermore, the associations we observed between
autistic symptomatology and ODD traits were small and mostly non-significant. This does not fit with the idea that the ODD
symptoms are merely epiphenomena of ASD. If that was the
case, we would expect oppositionality to intensify as autistic
symptomatology increased. Other investigations of ODD in
ASD (Guttmann-Steinmetz et al. 2009; Gadow et al. 2008a)
also found more similarities than differences between the
presentation of ODD in autistic and non-autistic samples. It
will be important in future to compare the neuropsychological
underpinnings of ODD traits in autistic and non-autistic samples to further investigate whether manifest oppositionality in
these two groups signifies the same underlying disorder. Also,
the use of confirmatory factor analysis to compare the structure
of ODD symptoms in people with and without ASD would
J Abnorm Child Psychol
provide additional information about the applicability of the
DSM-5 tripartite model to autistic populations.
To our knowledge, this is the first study to measure vindictiveness (i.e., vindictive and spiteful behaviour) in people with
ASD. Our findings suggest that this construct should receive
more attention in future. In both parent and teacher data,
around a fifth of our participants were described as definitely
showing vindictiveness; and this characteristic emerged as the
most consistent and strongest predictor of conduct disorder
symptoms. In particular, it was the only variable independently
associated with aggressive behaviour. Evolutionary psychologists have shown some interest in vindictiveness and spite,
conceptualising them as motivating adaptive social behaviours
that enforce moral codes through punishing unfair acts (Tooby
and Cosmides 2008). In our clinical experience, many children
with ASD have a heightened and somewhat rigid sense of
fairness, and can have difficulty accepting what they perceive
to be an injustice. It will be interesting to test to what extent
reported acts of vindictiveness and spite in our sample arise
from this characteristic; and by extension what role moral
inflexibility plays in aggressive behaviour in ASD.
Our findings highlight the importance of assessing AIS in
young people with ASD. Symptoms of AIS were common in
our clinical ASD sample and were associated with internalising
difficulties. People with ASD often struggle to understand and
articulate their feelings, which can make their anxiety and low
mood hard to detect (Hill et al. 2004). By contrast symptoms of
anger and irritability are usually directly observable and obvious. The presence of anger and irritability in children with
ASD should alert clinicians to the need for careful assessment
of anxiety and mood problems.
Why was AIS so common in our autistic sample? Leibenluft
(2011) has proposed a model of irritability in which rigidity,
dysregulated attention and poor emotional processing play a
role in both increasing the likelihood of experiencing frustration and decreasing the capacity to manage frustration. We note
that rigidity (Ozonoff et al. 2004), attention problems (Gadow
et al. 2005) and emotion processing deficits (Hobson et al.
1988) are all common in people with ASD, and this may go
some way to explaining the high rates of AIS in our sample. It
will be interesting to test Leibenluft’s model of irritability in
ASD by examining whether variability in rigidity, attention
problems and emotional processing predicts elevated AIS in
people with an autistic disorder.
The association between AIS and internalising difficulties
that we observed is consistent with the new DSM-5 diagnosis of temper deregulation disorder with dysphoria (TDDD),
which is characterised by irritability and negative affect.
Simonoff and colleagues (2012) recently examined ‘severe
mood problems’ (a construct very similar to TDDD) in their
clinically ascertained ASD cohort, and also observed an
association between irritability and low mood in ASD.
Given our findings and those of Simonoff and colleagues
(2012), it is likely that TDDD as described in DSM-5 will be
a valuable construct for characterising the nature of mood
disturbance in ASD. Studies that apply formal criteria for
TDDD in ASD are required.
Our cross-sectional analyses raise the question of whether
the adoption of the three dimension model of oppostionality
could improve the prognostic accuracy of ODD traits found in
people with ASD. This has been the case in non-autistic
populations (Stringaris and Goodman 2009b). Follow-up studies are required to test whether longitudinal findings in nonASD populations about AIS, ADB and vindictiveness apply to
people with ASD. A study of 135 adults with ASD who had
been diagnosed in childhood discovered that all psychopathology that emerged in adulthood involved disordered affect, with
depression and obsessive compulsive disorder being the most
common problems (Hutton et al. 2008). Likewise, comparison
of cross-sectional child and adult studies suggests that the
transition to adulthood in ASD is associated with an increased
risk of depression and obsessive compulsive disorder, as these
are diagnosed more often in adult compared to childhood autistic samples. In non-autistic people, AIS in childhood is a risk
factor for the subsequent development of affective and anxiety
disorders (Burke 2012; Stringaris et al. 2012). Furthermore,
temper outbursts, a key symptom of AIS, are highly prevalent
in obsessive compulsive disorder (OCD; Krebs et al. 2012). We
hypothesise that children with ASD who score high on AIS will
be at the greatest risk of developing mood and anxiety disorders,
including OCD, in adulthood. If longitudinal investigations
found this to be the case, it would promote the identification
of high-risk groups amongst young people with ASD, and
would suggest measures prophylactic against the development
of comorbidity in adolescence and adulthood.
Our findings are relevant to the development of treatments
for young people with ASD. ODD symptoms are common in
this population (Simonoff et al. 2008), and the risk they pose
for later aggression is amplified by co-occurring autistic
social impairments (Mandy et al. 2013). As such, there is a
pressing need for ODD treatments of proven effectiveness
for people with ASD. Our observations of commonalities
between oppositionality in people with and without ASD
suggest the value of testing existing evidence-based treatments for ODD in ASD samples. Interventions that treat
conduct problems by teaching parents to change their responses to their child’s problem behaviours (‘contingency
management’) are amongst the most effective in child psychology (Scott 2008). Such treatments should be evaluated
experimentally in young people with ASD and ODD, with
due consideration of whether autistic difficulties, such as
rigidity and empathy deficits, moderate treatment response.
Furthermore, our findings suggest a need for treatments for
AIS in ASD. Currently there is no basis for evidence-based
J Abnorm Child Psychol
practice in this area, although promising interventions have
been manualised (e.g., Attwood 2007) making them amenable for evaluation using randomised controlled trials.
The study we report had several strengths, including the use
of multiple informants, standardised measures administered by
experienced clinicians and robust statistical analyses to test
focused hypotheses that were grounded in previous literature.
Nevertheless, its findings should be considered in relation to the
following limitations. Our data were cross-sectional and so are
not well suited to testing causal hypotheses. Also, the sample
comprised individuals with fluent language who were in mainstream education, and so is not representative of the full population of people with ASD. As such, whether or not our
findings can be generalised to people with low-functioning
ASD, who comprise up to 50 % of those with an ASD diagnosis (Baird et al. 2006), remains an open question for further
empirical investigation. Our dimensional measures of conduct
problems were based on DSM-IV symptom checklists. This
allowed us to compare our results to relevant studies of nonautistic people, and generally ensured our measures had content
validity, in that they covered the full range of ODD and CD
symptoms. Nevertheless, our study would have benefitted
from the use of fine-grained measures with more items. This
is especially true for our teacher data, and for the vindictiveness construct, which was measured by a single item in the
current study, reflecting the fact that only one ODD diagnostic criterion in DSM-IV (and indeed DSM-5) covers vindictiveness. Furthermore, our measure of internalising problems
did not distinguish low mood and anxiety. It will be useful in
future to measure these constructs independently, to test
whether they are differentially associated with irritability.
Conclusions
Our findings highlight the importance of measuring ODD traits
in the clinical assessment of ASD. There was considerable
variability amongst our participants in terms of ODD symptom
severity, and this variability contained important clinical information about an individual’s tendency to have difficulties with
ADHD symptoms, internalising and conduct. Furthermore,
despite their high intercorrelations, AIS, ADB and vindictiveness dimensions of oppositionality had distinct associations
with concurrent psychopathology, suggesting the need to assess
these as separate constructs. AIS in ASD is part of a picture of
comorbidity involving anxiety and low mood. By contrast
ADB and vindictiveness associate with externalising problems,
such as ADHD and disruptive behaviour.
Acknowledgements The authors wish to thank Richard Warrington
whose 3Di software made this study possible, as well as the anonymous
reviewers for their helpful suggestions and comments.
References
American Psychiatric Association. (2000). Diagnostic and statistical
manual (4th ed., text rev.). Washington: Author.
American Psychiatric Association. (2013). Diagnostic and statistical
manual (5th ed.). Washington: Author.
Attwood, T. (2007). Exploring feelings: Cognitive behaviour therapy to
manage anger. Arlington: Future Horizons Inc.
Baird, G., Simonoff, E., Pickles, A., Chandler, S., Loucas, T., Meldrum,
D., & Charman, T. (2006). Prevalence of disorders of the autism
spectrum in a population cohort of children in south Thames: The
special needs and autism project (SNAP). Lancet, 368, 210–5.
Burke, J. D. (2012). An affective dimension within oppositional defiant
disorder symptoms among boys: personality and psychopathology
outcomes into early adulthood. Journal of Child Psychology and
Psychiatry, and Allied Disciplines, 53, 1176–1183.
Burke, J., Loeber, R., & Birmaher, B. (2002). Oppositional defiant
disorder and conduct disorder: A review of the past 10 years, part
II. Journal of the American Academy of Child and Adolescent
Psychiatry, 41, 1275–93.
Gadow, K. D., DeVincent, C. J., & Drabick, D. A. (2008a).
Oppositional defiant disorder as a clinical phenotype in autism
spectrum disorder. Journal of Autism and Developmental Disorders,
38, 1302–10.
Gadow, K. D., DeVincent, C. J., Pomeroy, J., & Azizian, A. (2005).
Comparison of DSM-IV symptoms in elementary school-age children with PDD versus clinic and community samples. Autism: the
international journal of research and practice, 9, 392–415.
Gadow, K. D., DeVincent, C. J., Pomeroy, J., & Azizian, A. (2004).
Psychiatric symptoms in preschool children with PDD and clinic
and comparison samples. Journal of Autism and Developmental
Disorders, 34, 379–393.
Gadow, K. D., DeVincent, C., & Schneider, J. (2008b). Predictors of
psychiatric symptoms in children with an autism spectrum disorder.
Journal of Autism and Developmental Disorders, 38, 1710–1720.
Goodman, A., & Goodman, R. (2009). Strengths and difficulties questionnaire as a dimensional measure of child mental health. Journal
of the American Academy of Child and Adolescent Psychiatry, 48,
400–403.
Goodman, R. (1997). The strengths and difficulties questionnaire: A
research note. Journal of Child Psychology and Psychiatry, and
Allied Disciplines, 38, 581–586.
Goodman, R. (2001). Psychometric properties of the strengths and
difficulties questionnaire. Journal of the American Academy of
Child and Adolescent Psychiatry, 40, 1337–1345.
Goodman, R., & Scott, S. (1999). Comparing the strengths and difficulties questionnaire and the child behavior checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24.
Guttmann-Steinmetz, S., Gadow, K. D., & DeVincent, C. J. (2009).
Oppositional defiant and conduct disorder behaviors in boys with
autism spectrum disorder with and without attention-deficit hyperactivity disorder versus several comparison samples. Journal of
Autism and Developmental Disorders, 39, 976–985.
Hill, E., Berthoz, S., & Frith, U. (2004). Brief report: cognitive processing of own emotions in individuals with autistic spectrum
disorder and in their relatives. Journal of Autism and Developmental
Disorders, 34, 229–235.
Hobson, R. P., Ouston, J., & Lee, A. (1988). Emotion recognition in
autism: Coordinating faces and voices. Psychological Medicine,
18, 911–923.
Hutton, J., Goode, S., Murphy, M., Le Couteur, A., & Rutter, M. (2008).
New-onset psychiatric disorders in individuals with autism.
Autism: the international journal of research and practice, 12,
373–390.
J Abnorm Child Psychol
Krebs, G., Bolhuis, K., Heyman, I., Mataix-Cols, D., Turner, C., &
Stringaris, A. (2012). Temper outbursts in paediatric obsessivecompulsive disorder and their association with depressed mood and
treatment outcome. Journal of Child Psychology and Psychiatry,
and Allied Disciplines, 54, 313–22.
Krieger, F. V., Polanczyk, V. G., Robert, G., Rohde, L. A., GraeffMartins, A. S., Salum, G., & Stringaris, A. (2013). Dimensions
of oppositionality in a Brazilian community sample: testing the
DSM-5 proposal and etiological links. Journal of the American
Academy of Child and Adolescent Psychiatry, 52, 389–400.
Lecavalier, L., Gadow, K. D., DeVincent, C. J., & Edwards, M. C.
(2009). Validation of DSM-IV model of psychiatric syndromes in
children with autism spectrum disorders. Journal of Autism and
Developmental Disorders, 39, 278–289.
Leibenluft, E. (2011). Severe mood dysregulation, irritability, and the
diagnostic boundaries of bipolar disorder in youths. The American
Journal of Psychiatry, 168, 129–142.
Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Jr., Leventhal, B. L.,
DiLavore, P. C., & Rutter, M. (2000). The autism diagnostic
observation schedule-generic: A standard measure of social and
communication deficits associated with the spectrum of autism.
Journal of Autism and Developmental Disorders, 24, 659–85.
Lugnegård, T., Hallerbäck, M. U., & Gillberg, C. (2011). Psychiatric
comorbidity in young adults with a clinical diagnosis of asperger
syndrome. Research in Developmental Disabilities, 32, 1910–1917.
Mandy, W., Skuse, D., Steer, C., St Pourcain, B., & Oliver, B. (2013).
Oppositionality and socio-emotional competence: Interacting risk
factors in the development of childhood conduct disorder symptoms. Journal of the American Academy of Child and Adolescent
Psychiatry, 52, 718–27.
Ozonoff, S., Cook, I., Coon, H., Dawson, G., Joseph, R. M., Klin, A., &
Wrathall, D. (2004). Performance on Cambridge neuropsychological test automated battery subtests sensitive to frontal lobe function in people with autistic disorder: Evidence from the collaborative programs of excellence in autism network. Journal of Autism
and Developmental Disorders, 34, 139–150.
Rowe, R., Costello, E. J., Angold, A., Copeland, W. E., & Maughan, B.
(2010). Developmental pathways in oppositional defiant disorder
and conduct disorder. Journal of Abnormal Psychology, 119, 726–
738.
Scott, S. (2008). Parenting programmes. In M. Rutter, D. Bishop, D.
Pine, S. Scott, J. Stevenson, E. Taylor, & A. Thapar (Eds.), Rutter’s
child and adolescent psychiatry (5th ed.). Oxford: Blackwell.
Simonoff, E., Jones, C. R. G., Baird, G., Pickles, A., Happé, F., & Charman,
T. (2013). The persistence and stability of psychiatric problems in
adolescents with autism spectrum disorders. Journal of Child
Psychology and Psychiatry, and Allied Disciplines, 54, 186–194.
Simonoff, E., Jones, C. R. G., Pickles, A., Happé, F., Baird, G., &
Charman, T. (2012). Severe mood problems in adolescents with
autism spectrum disorder. Journal of Child Psychology and
Psychiatry, and Allied Disciplines, 53, 1157–1166.
Simonoff, E., Pickles, A., Charman, T., Chandler, S., Loucas, T., &
Baird, G. (2008). Psychiatric disorders in children with autism
spectrum disorders: prevalence, comorbidity, and associated
factors in a population-derived sample. Journal of the American
Academy of Child and Adolescent Psychiatry, 47, 921–929.
Skuse, D., Warrington, R., Bishop, D., Chowdhury, U., Lau, J., Mandy,
W., & Place, M. (2004). The developmental, dimensional and
diagnostic interview (3di): A novel computerized assessment for
autism spectrum disorders. Journal of the American Academy of
Child and Adolescent Psychiatry, 43, 548–558.
StataCorp. (2011). Stata statistical software: Release 12. College
Station, TX: StataCorp LP.
Sterba, S. K., Copeland, W., Egger, H. L., Jane Costello, E., Erkanli, A., &
Angold, A. (2010). Longitudinal dimensionality of adolescent psychopathology: Testing the differentiation hypothesis. Journal of Child
Psychology and Psychiatry, and Allied Disciplines, 51, 871–884.
Stringaris, A., & Goodman, R. (2009a). Three dimensions of oppositionality
in youth. Journal of Child Psychology and Psychiatry, and Allied
Disciplines, 50, 216–223.
Stringaris, A., & Goodman, R. (2009b). Longitudinal outcome of youth
oppositionality: Irritable, headstrong, and hurtful behaviors have
distinctive predictions. Journal of the American Academy of Child
and Adolescent Psychiatry, 48, 404–412.
Stringaris, A., Zavos, H., Leibenluft, E., Maughan, B., & Eley, T. C.
(2012). Adolescent irritability: Phenotypic associations and genetic
links with depressed mood. The American Journal of Psychiatry,
169, 47–54.
Tackett, J., Krueger, R., Iacono, W., & McGue, M. (2005). Symptombased subfactors of DSM-defined conduct disorder: Evidence for
etiologic distinctions. Journal of Abnormal Psychology, 114,
483–7.
Tooby, J., & Cosmides, L. (2008). The evolutionary psychology
of the emotions and their relationship to internal regulatory
variables. In M. Lewis, J. Haviland-Jones, & L. Barrett (Eds.),
Handbook of emotions (3rd ed., pp. 114–137). New York: Guilford.
van Steensel, F. J. A., Bögels, S. M., & Perrin, S. (2011). Anxiety
disorders in children and adolescents with autistic spectrum disorders: A meta-analysis. Clinical Child and Family Psychology Review,
14, 302–317.