Download Table 1. Aggregate comparisons of CAOS scores Average percent

Table 1. Aggregate comparisons of CAOS scores
Average percent correct
Sample size
National sample (NT)
Hope sample Fall 2007 (HT)
Hope sample Fall 2009 (HR)
763
198
202
Pre-test
Mean (SD)
1
44.9 (- )
48.4 (11.2)
44.7 (9.3)
Post-test
Mean (SD)
1
54.0 (- )
57.2 (11.8)
55.7 (11.8)
Difference in average percent
correct
Mean (SD)
9.1 (12.0)
8.9 (9.9)
11.0 (11.3)
1. Standard deviations for the pre and posttest were not available, but the standard deviation of the difference was recorded in delMas et al.
(2007)
Table 2. Items for which students in the HR cohort learned significantly more than the HT cohort
% of Students Correct
Item
number
on CAOS
Item Description (Topic)
1
Cohort
Pretest
Posttest
Difference
McNemar’s
Test p-value
Cohort
p-value2
aOR (95%CI)2
7
Understanding of the purpose of
randomization in an experiment (Data
collection and design)
NT
HT
HR
8.5
4.6
3.5
12.3
9.7
20.8
3.8
5.1
17.3
0.013
0.076
<0.001
0.001
0.5 (0.3, 0.8)**
0.4 (0.2, 0.7)**
1.0
19
Understanding that low p-values are
desirable in research studies (Tests of
significance)
NT
HT
HR
49.9
56.9
56.9
68.5
85.6
96.0
18.6
28.7
39.1
<0.001
<0.001
<0.001
<0.001
0.1 (0.0, 0.2)***
0.2 (0.1,0.5)**
1.0
23
Understanding that no statistical significance
does not guarantee that there is no effect
(Tests of significance)
NT
HT
HR
63.1
66.2
65.2
64.4
72.7
85.1
1.3
6.5
19.9
0.630
0.130
<0.001
<0.001
0.3 (0.2, 0.5)***
0.4 (0.3, 0.8)**
1.0
25
Ability to recognize a correct interpretation
of a p-value (Tests of significance)
NT
HT
HR
46.8
36.1
42.3
54.5
41.0
60.0
7.7
4.9
17.7
0.005
0.402
0.002
<0.001
0.8 (0.6, 1.1)
0.5 (0.3, 0.7)***
1.0
26
Ability to recognize an incorrect
interpretation of a p-value. Specifically,
probability that a treatment is not effective.
(Tests of significance)
NT
HT
HR
53.1
59.8
58.9
58.6
68.6
79.7
5.5
8.8
20.8
0.044
0.085
<0.001
<0.001
0.4 (0.3, 0.5)***
0.6 (0.3, 0.9)*
1.0
37
Understanding of how to simulate data to
find the probability of an observed value
(Probability)
NT
HT
HR
20.4
20.0
20.0
19.5
20.0
32.2
-0.9
0.0
12.2
0.713
1.000
0.009
0.001
0.5 (0.4, 0.7)***
0.5 (0.3, 0.8)**
1.0
1.
2.
NT= National sample with the Traditional curriculum, HT= Hope sample (2007) with the traditional curriculum, HR= Hope sample (2009) with the new
curriculum
Results from a logistic regression model predicting post-test (right/wrong) by curriculum, controlling for pre-test right/wrong. Cohort p-value gives the
overall p-value for the cohort term, and aOR gives the adjusted odds ratio (and corresponding 95% CI) comparing each curriculum to the new
randomization based curriculum. *p<0.05, **p<0.01 and ***p<0.001.
Table 3. Items for which students in the HR cohort learned significantly less than the HT cohort
% of Students Correct
Item
number
on CAOS
Item Description (Topic)
Ability to correctly estimate and compare
standard deviations for different
histograms. (Descriptive statistics)
14
1.
2.
1
Cohort
NT
HT
HR
Pretest
34.3
44.8
36.3
Posttest
Difference
McNemar’s
Test p-value
51.7
70.8
48.5
17.4
26.0
12.2
<0.001
<0.001
0.006
Cohort
p-value2
aOR (95%CI)2
<0.001
1.2 (0.9, 1.7)
2.5 (1.6, 3.9)***
1.0
NT= National sample with the Traditional curriculum, HT= Hope sample (2007) with the traditional curriculum, HR= Hope sample (2009) with the new
curriculum
Results from a logistic regression model predicting post-test (right/wrong) by curriculum, controlling for pre-test right/wrong. Cohort p-value gives the
overall p-value for the cohort term, and aOR gives the adjusted odds ratio (and corresponding 95% CI) comparing each curriculum to the new
randomization based curriculum. *p<0.05, **p<0.01 and ***p<0.001.
Table 4. Learning differences by topic1
Topic
Data collection
and Design
Descriptive
statistics
Total
numb
er of
items
4
New curriculum performed
better than Hope traditional
(Table 2)
New curriculum performed
worse than Hope traditional
(Table 3)
Other significant
differences between
samples
(Table A1)
7
3
No significant differences
between groups
(Table A2)
22, 24, 38
14
15, 18
Graphical
representations
Boxplots
9
6
1, 3, 4, 5, 11, 12, 13, 33
4
2
8,9,10
Bivariate data
3
39
20, 21
Probability
2
Sampling
variability
5
17
16,32,34,35
Confidence
Intervals
4
28,29,30
31
Tests of
significance
6
37
19, 23, 25, 26
1. CAOS item numbers are in the table
36
27,40
Table A1. Other items with significant differences between cohorts
% of Students Correct
Item
number
on CAOS
Item Description (Topic)
1
Cohort
Pretest
Posttest
Difference
McNemar’s
Test p-value
Cohort
p-value2
aOR (95%CI)2
2
Ability to recognize two different graphical
representations of the same data (boxplot
and histogram) (Boxplots)
NT
HT
HR
45.5
49.0
53.0
56.3
66.7
72.3
10.8
17.7
19.3
<0.001
<0.001
<0.001
<0.001
0.5 (0.4, 0.7)***
0.8 (0.5, 1.2)
1.0
6
Understanding that to properly describe the
distribution of a quantitative variable, a
graph like a histogram is needed (Graphical
Representations)
NT
HT
HR
15.1%
9.7%
5.9%
25.2%
15.4%
11.4%
10.10%
5.70%
5.50%
<0.001
0.052
0.061
0.001
2.2 (1.4, 3.6)**
1.3 (0.7, 2.4)
1.0
17
Understanding of expected patterns in
sampling variability (Sampling variability)
NT
HT
HR
42.8
54.1
40.1
50.3
66.2
57.9
7.5
12.1
17.8
<0.001
0.002
<0.001
0.001
0.7 (0.5, 0.9)*
1.2 (0.8, 1.9)
1.0
28
Ability to detect a misinterpretation of a
confidence level (the percentage of sample
data between confidence limits)
(Confidence intervals)
NT
HT
HR
48.4
48.7
52.0
43.2
50.3
58.7
-5.2
1.6
6.7
<0.001
0.679
0.762
<0.001
0.5 (0.4, 0.7)***
0.7 (0.5, 1.1)
1.0
29
Ability to detect a misinterpretation of a
confidence level (percentage of population
data values between confidence limits)
(Confidence Intervals)
NT
HT
HR
32.6
34.9
35.3
67.6
50.8
58.4
35.0
15.9
23.1
<0.001
0.001
<0.001
<0.001
1.5 (1.1, 2.1)*
0.7 (0.5, 1.1)
1.0
30
Ability to detect a misinterpretation of a
confidence level (percentage of all possible
sample means between confidence limits)
(Confidence Intervals)
NT
HT
HR
31.4
35.2
35.1
44.2
26.3
32.8
12.8
-8.9
-2.3
<0.001
0.085
0.665
<0.001
1.6 (1.1, 2.3)**
0.7 (0.5, 1.1)
1.0
36
Understanding of how to calculate
appropriate ratios to find conditional
probabilities using a table of data
NT
HT
HR
52.7
49.7
46.8
53.0
71.8
71.1
0.3
22.1
24.3
0.955
<0.001
<0.001
<0.001
0.4 (0.3, 0.6)***
1.0 (0.7, 1.6)
1.0
(Probability)
39
Understanding of when it is not wise to
extrapolate using a regression model
(Bivariate data)
1.
2.
NT
HT
HR
17.9
9.7
15
24.5
11.8
8.9
6.6
2.1
-6.1
0.001
0.618
0.066
<0.001
3.5 (2.0, 5.9)***
1.5 (0.8, 2.9)
1.0
NT= National sample with the Traditional curriculum, HT= Hope sample (2007) with the traditional curriculum, HR= Hope sample (2009) with the new
curriculum
Results from a logistic regression model predicting post-test (right/wrong) by curriculum, controlling for pre-test right/wrong. Cohort p-value gives the
overall p-value for the cohort term, and aOR gives the adjusted odds ratio (and corresponding 95% CI) comparing each curriculum to the new
randomization based curriculum. *p<0.05, **p<0.01 and ***p<0.001.
Table A2. Items without significant differences between cohorts
% of Students Correct
Item
number
on CAOS
Item Description (Topic)
1
Cohort
Pretest
Posttest
Difference
McNemar’s
test p-value
Cohort
p-value2
aOR (95%CI)2
1
Ability to describe and interpret the overall
distribution of a variable as displayed in a
histogram (Graphical Representations)
NT
HT
HR
71.1
75.9
68.3
73.6
78.5
80.2
2.5
2.6
11.9
0.291
0.597
0.012
0.088
0.7 (0.5, 1.0)
0.9 (0.5, 1.4)
1.0
3
Ability to visualize and match a histogram
to a description (negative skewed
distribution for scores on an easy quiz)
(Graphical representations)
NT
HT
HR
56.7
71.3
60.9
73.2
86.7
76.7
16.5
15.4
15.8
<0.001
<0.001
<0.001
0.019
0.9 (0.6, 1.3)
1.7 (1.0, 3.0)
1.0
4
Ability visualize and match a histogram to a
description of a variable (bell-shaped
distribution) (Graphical Representations)
NT
HT
HR
48.0
53.6
41.3
63.1
63.1
60.9
15.1
9.5
19.6
<0.001
0.027
<0.001
0.931
1.0 (0.7, 1.4)
1.0 (0.6, 1.5)
1.0
5
Ability to visualize and match a histogram
to a description of a variable (uniform
distribution) (Graphical representations)
NT
HT
HR
55.9
68.6
55.9
71.1
81.5
68.3
15.2
12.9
12.4
<0.001
<0.001
0.004
0.066
1.2 (0.8, 1.7)
1.8 (1.1, 3.0)*
1.0
8
Ability to determine which of two boxplots
represents a larger standard deviation
(Boxplots)
NT
HT
HR
54.7
52.8
56.7
59.2
62.6
48.0
4.5
9.8
-8.7
0.068
0.025
0.082
0.004
1.6 (1.2, 2.2)**
1.9 (1.2, 2.8)**
1.0
9
Understanding that boxplots do not provide
accurate estimates for percentages of data
above or below values except for the
quartiles (Boxplots)
NT
HT
HR
23.3
19.6
10.0
26.6
23.1
23.4
3.3
3.5
13.4
0.114
0.360
<0.001
0.742
1.0 (0.7, 1.5)
0.9 (0.5, 1.4)
1.0
10
Understanding of the interpretation of a
median in the context of boxplots
(Boxplots)
NT
HT
HR
19.6
21.0
17.3
28.3
33.8
33.2
8.7
12.8
15.9
<0.001
<0.001
<0.001
0.197
0.8 (0.5, 1.1)
1.0 (0.7, 1.6)
1.0
11
Ability to compare groups by considering
where most of the data are, and focusing on
distributions as single entities (Graphical
representations)
NT
HT
HR
88.0
93.3
89.6
88.2
94.9
92.5
0.2
1.6
2.9
0.928
0.629
0.362
0.027
0.6 (0.3, 1.1)
1.4 (0.6, 3.2)
1.0
12
Ability to compare groups by comparing
differences in averages (Graphical
representations)
NT
HT
HR
85.3
89.2
85.1
85.8
88.7
85.6
0.5
-0.5
0.5
0.804
1.00
1.00
0.716
1.0 (0.6, 1.6)
1.2 (0.7, 2.3)
1.0
13
Understanding that comparing two groups
does not require equal sample sizes in each
group, especially if both sets of data are
large (Graphical Representations)
NT
HT
HR
61.8
63.1
55.2
73.5
79
71.8
11.7
15.9
16.6
<0.001
<0.001
<0.001
0.302
1.0 (0.7, 1.4)
1.3 (0.8, 2.2)
1.0
15
Ability to correctly estimate standard
deviations for different histograms.
(Descriptive statistics)
NT
HT
HR
38.3
42.3
41.8
46.9
51.3
48.5
8.6
9.0
6.7
<0.001
0.053
0.203
0.688
1.0 (0.7, 1.3)
1.1 (0.7, 1.7)
1.0
16
Understanding that statistics from small
samples vary more than statistics from large
samples (Sampling variability)
NT
HT
HR
22.8
23.1
21.8
31.9
42.1
29.4
9.1
19.0
7.6
<0.001
<0.001
0.026
0.008
1.1 (0.7, 1.6)
1.9 (1.2, 2.9)**
1.0
18
Understanding the meaning of variability in
the context of repeated measurements, and
in a context where small variability is
desired (Descriptive statistics)
NT
HT
HR
80.6
86.7
82.6
80.6
87.7
78.7
0.0
1.0
-3.9
1.000
0.856
0.280
0.084
1.2 (0.8, 1.8)
1.9 (1.1, 3.3)*
1.0
20
Ability to match a scatterplot to a verbal
description of a bivariate relationship
(Bivariate Data)
NT
HT
HR
90.5
95.4
92.1
92.5
92.8
94.5
2.0
-2.6
2.4
0.159
0.383
0.541
0.644
0.7 (0.4, 1.4)
0.7 (0.3, 1.6)
1.0
21
Ability to correctly describe a bivariate
relationship shown in a scatterplot when
there is an outlier (influential point)
(Bivariate data)
NT
HT
HR
73.6
80.5
83.7
89.7
10.1
9.2
<0.001
0.010
0.004
0.165
1.1 (0.7, 1.6)
1.7 (0.9, 3.1)
1.0
71.1
82.7
11.6
22
Understanding that correlation does not
imply causation (Data collection and
design)
NT
HT
HR
54.6
52.1
44.1
52.6
54.4
61.9
-2.0
2.3
17.8
0.404
0.640
<0.001
0.011
0.6 (0.4, 0.8)**
0.7 (0.4, 1.0)
1.0
24
Understanding that an experimental design
with random assignment supports causal
inference (Data collection and design)
NT
HT
HR
58.5
64.6
56.3
59.5
65.5
59.4
1.0
0.9
3.1
0.731
1.000
0.505
0.441
0.9 (0.7, 1.3)
1.2 (0.8, 1.8)
1.0
27
Ability to recognize an incorrect
interpretation of a p-value. Specifically, as
the probability a treatment is effective.
(Tests of significance)
NT
HT
HR
42.3
37.1
52.7
47.7
10.4
10.6
<0.001
0.027
0.073
0.128
1.3 (1.0, 1.8)
1.1 (0.7, 1.6)
1.0
35.8
44.6
8.8
31
Ability to correctly interpret a confidence
interval (Confidence intervals)
NT
HT
HR
47.1
46.2
41.8
74.3
80.5
67.8
27.2
34.3
26.0
<0.001
<0.001
<0.001
0.017
1.4 (1.0, 1.9)
2.0 (1.2, 3.1)**
1.0
32
Understanding of how sampling errors are
used to make an informal inference about a
sample mean (Sampling variability)
NT
HT
HR
16.9
14.4
17.4
17.1
8.2
13.4
0.2
-6.2
-4.0
0.941
0.073
0.302
0.009
1.3 (0.8, 2.1)
0.6 (0.3, 1.1)
1.0
33
Understanding that a distribution with the
median larger than mean is most likely
skewed to the left (Graphical
representations)
NT
HT
HR
41.5
42.6
39.7
43.6
-1.8
1.0
0.511
0.941
0.755
0.312
1.2 (0.8, 1.6)
1.4 (0.9, 2.1)
1.0
37.6
35.8
-1.8
34
Understanding the law of large numbers for
a large sample by selecting an appropriate
sample from a population given the sample
size (Sampling variability)
NT
HT
HR
55.3
65.6
53.5
65.2
70.3
55.9
9.9
4.7
2.4
<0.001
0.368
0.649
0.020
1.5 (1.1, 2.0)*
1.7 (1.1, 2.6)**
1.0
35
Ability to select an appropriate sampling
distribution for a population and sample
size (Sampling variability)
NT
HT
HR
34.5
37.6
30.5
44.2
50.5
43.1
9.7
12.9
12.6
<0.001
0.013
0.010
0.341
1.0 (0.7, 1.4)
1.3 (0.9, 1.9)
1.0
Understanding of the factors that allow a
sample of data to be generalized to the
population (Data collection and design)
38
40
1.
2.
NT
HT
HR
26.0
25.1
20.8
37.9
34.4
29.5
11.9
9.3
8.7
<0.001
0.038
0.033
0.143
1.4 (1.0, 2.0)
1.2 (0.8, 1.9)
1.0
Understanding of the logic of a significance
NT
41.9
52
10.1
0.820
0.9 (0.7, 1.3)
<0.001
test when the null hypothesis is rejected
HT
36.4
53.3
16.9
1.0 (0.7, 1.5)
<0.001
(Tests of significance)
HR
40.6
53.5
12.9
0.010
1.0
NT= National sample with the Traditional curriculum, HT= Hope sample (2007) with the traditional curriculum, HR= Hope sample (2009) with the new
curriculum
Results from a logistic regression model predicting post-test (right/wrong) by curriculum, controlling for pre-test right/wrong. Cohort p-value gives the
overall p-value for the cohort term, and aOR gives the adjusted odds ratio (and corresponding 95% CI) comparing each curriculum to the new
randomization based curriculum. *p<0.05, **p<0.01 and ***p<0.001.