Download Handout File - OSPI Educational Technology Moodle

WASHINGTON EDUCATIONAL SERVICE DISTRICT NETWORK
Data Coach STATS Boot Camp
Data Coach Stats Boot Camp is designed to provide a brief
survey of statistical concepts, with an emphasis on illustrating
effect sizes and the foundations of statistical inference to help in
making evidence based decisions for educational practices.
Tentative Agenda
Time
9:00 – 9:15
9:15 - 9:45
10:15 – 10:30
10:30 – 11:45
11:45 – 12:30
12:30 – 1:45
1:45 – 2:00
2:00 – 3:30
3:30- 4:00
Activity
Welcome
Data Harvesting
 Locating
 Accessing
Descriptive Statistics
 Defining Terms
 Measures of Central Tendency
Break
Frequency Distribution
Graphing Data
Measures of Dispersion
Standard Scores
The Normal Distribution
Lunch
Evidence-Based Harvesting
Reviewing the Literature
Discovering Terms
Break/Reading
Effect Size + Standard Deviation
Percentile + Improvement Index
Correlation + Coefficient of Determination
Closing
Facilitator
Sue Furth & Sue Feldman
Todd Johnson
Tara Richerson
Todd Johnson
Tara Richerson
Todd Johnson
Tara Richerson
Todd Johnson
Tara Richerson
Sue Feldman
DATA HARVESTING
U.S. Census Bureau
http://www.census.gov/
Ed Data Express:
http://www.eddataexpress.ed.gov/
ED.gov Civil Rights Data Collection:
http://ocrdata.ed.gov/
Revised: May 5, 2017
2
National Center for Education Statistics
Data Tools:
http://nces.ed.gov/datatools/
P-20 high school feedback reports:
http://erdcdata.wa.gov/
OSPI Data Reports
http://www.k12.wa.us/DataAdmin/default.aspx
Revised: May 5, 2017
3
Washington Query
Washington Query is a data management and
reporting system that allows you to analyze data and
create reports for the following assessments: HSPE.
MSP. HSPB. MSPB. PORT. DAPE . WASL. WABA. and
DAWL.
https://eds.ospi.k12.wa.us
COMING SOON!!!
Statewide Longitudinal Data System (SLDS)
K–12 Data and Reports
http://www.k12.wa.us/Data/
Washington Achievement Data Explorer (WADE): Student Performance Across WA State http://www.cedr.us/WADE.html
The BERC Group: College Tracking Data Services http://www.collegetracking.com/
WA State Advanced Placement(AP) Trend Data http://www.k12.wa.us/AdvancedPlacement/trenddata.aspx
WA K-12 Enrollment, Staffing, and Finance displayed here are based on data from the Office of the Superintendent of
Public Instruction School Apportionment and Financial Services. Reports provide school district-specific detail for
enrollment, staffing, revenue, and expenditure categories. http://www.fiscal.wa.gov/k12.aspx
WA Kids Count Data Center: Access profiles for many WA locations; rankings, maps, or trend graphs by topic; and raw
data. Includes over 100 measures of child well-being, including the community-level data.
http://datacenter.kidscount.org/
WA Healthy Youth Survey is Washington State's school based survey that measures health risk behaviors that contribute
to morbidity, mortality, and social problems among youth. AskHYS.net is an online system designed to help you access,
analyze and disseminate. http://www.askhys.net/layout.asp?page=reports/FactSheets
http://reportcard.ospi.k12.wa.us/
http://reportcard.ospi.k12.wa.us/DataDownload.aspx
Revised: May 5, 2017
4
EXERCISE: MATCHING
STATS TERMS FOR THE DAY
Number Jumbled
1
2
Descriptive statistics
Effect size
Answer
(Number)
Definition
A number that indicates the strength and direction of
the statistical association between two or more
variables.
A score in a set of scores or a frequency distribution
that is typical or representative of all the scores.
Measures of central tendency are the mean, median
and mode.
Standard score
The bell-shaped curve that results from the graph of a
normal frequency distribution.
Correlation coefficient
Methods and rules for organizing and interpreting
quantitative observations.
Inferential statistics
The percent of participants who score at or below a
particular score in a frequency distribution
6
Central Tendency
Statistics used to describe, organize and summarize
data.
7
Normal curve
8
Percent
9
Statistics
3
4
5
10
Intervention
11
Percentile
12
Frequency
distribution
The proportion of participants who obtain a particular
score in a frequency distribution.
A score that transforms an original or raw score into
standard deviation units in order to locate the score’s
position within a frequency distribution.
The frequency of occurrence of scores in a set.
The degree to which a practice, program or policy has
an effect based on research results, measured in
standard deviation units.
Statistics used to make inferences about a population
based on the scores obtained from a sample.
A procedure, technique or strategy that is designed to
modify an ongoing process
Revised: May 5, 2017
5
THE NORMAL CURVE, PERCENTILES, AND SELECTED STANDARD SCORES
Revised: May 5, 2017
6
EXERCISE (Remind me to show you accessing Reports)
NWEA MAP DATA
Descriptive Statistics
Grade
Measures of Central Tendency
Reading
Math
Mean Median Mode SD
Mean Median Mode SD
6
7
8
Gender
Male
Female
Any there other measure(s) you need or feel are missing?
STOP!!!
Discussion
EXERCISE:
STATE ASSESSMENT DATA
1. For Washington State, how many SCHOOLS had students taking assessments last year for?
Grade 6 ________
Grade 7 ________
Grade 8 ________
2. For Washington State, how many STUDENTS were assessed in Reading for the following grade levels?
Grade 6 ________
Grade 7 ________
Grade 8 ________
3. Which ESD had the most Math Total MSP_HSPE Tested?
4. Which ESD had fewer than 23,000 students in Reading Total MSP_HSPE Tested?
5. Write your own question about the data and answer it here
QUESTION:
Answer:
STOP!!!
Discussion
Revised: May 5, 2017
7
MATH - Frequency Distributions
Identify two ESD’s you will work with
ESD 1
MSP Level
L1
L2
L3
L4
Frequency
Relative
Frequency
ESD 2
(RF= fi/n)
MSP Level
L1
L2
L3
L4
Relative
Frequency
Frequency
(RF= fi/n)
Totals
Complete your work on another Tab in the Spreadsheet if needed
STOP!!!
Discussion
Write down three questions you can think of that these data could answer for you or others?
Reflecting back on what have been done so far, what other data would you like to explore or have to help
you in your data coaching?
Revised: May 5, 2017
8
Standard Score: A way of representing performance on a test or some other measure so that persons familiar
with the standard score know immediately how well the person did relative to others taking the same test.
EXERCISE
EasyCBM Data
Descriptive Statistics
Grade
Mean
Measures of Central Tendency
Reading
Math
Median Mode SD
Mean Median Mode SD
6
7
8
Gender
Male
Female
STOP!!!
Discussion
Answer the following based on your data
N
Range (n)
Low High
Low
Risk Level(n)
Some High
Reading_Comp_Score
Math_Score
Vocabulary_Score
How did you address the missing student scores?
What Percentile Score ranges would you recommend and why?
Would these recommendations be different based on Grade Level and why?
STOP!!!
Discussion
Revised: May 5, 2017
9
EVIDENCE-BASED HARVESTING
Best Evidence Encyclopedia (BEE)
http://www.bestevidence.org/
SAMHSA's National Registry o
Evidence-based Programs
and Practices
SAMHSA's National Registry of
Evidence-based Programs and
Practices
http://www.nrepp.samhsa.gov/
EFFECT SIZE: The degree to which a practice, program or policy has an effect based on
research results, measured in standard deviation units.
(Effect size is also referred to as practical significance.)
Revised: May 5, 2017
10
National Center on Response to
Intervention
http://www.rti4success.org/
Meta-Analysis Database
Understanding Marzano Research
Laboratory's Action Research MetaAnalysis Database
http://www.marzanoresearch.com/research/meta_analysis_database.aspx
Revised: May 5, 2017
11
What Works Clearinghouse (WWC)
http://ies.ed.gov/ncee/wwc/
ESD STATS TRANSLATOR EXCERCISES
WWC Review of the Report “Enhancing the Effectiveness of Special Education Programming for
Children with Attention Deficit Hyperactivity Disorder Using a Daily Report Card”
http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=10008
Study About
Statistical
Significance Found
Grades
Examined
Topic
Effect Size(ES)
Improvement Index
Percentile
Increase
Standard
Deviation
Coefficient of
Determination
WWC Review of the Report “Effects of Problem Based Economics on High School Economics Instruction”
http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=10006
Study About
Statistical
Significance Found
Grades
Examined
Topic
Effect Size(ES)
Improvement Index
Percentile
Increase
Standard
Deviation
Coefficient of
Determination
WWC Review of the Report “Accommodations for English Language Learner Students: The Effect of Linguistic
Modification of Math Test Item Sets”
http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=10004
Study About
Statistical
Significance Found
Grades
Examined
Topic
Effect Size(ES)
Improvement Index
Percentile
Increase
Standard
Deviation
Coefficient of
Determination
Revised: May 5, 2017
12
Effective Research-Based
High-Yield Instructional Strategies
Marzano, et. al.1 (2000) identified ten research-based, effective instructional strategies that cut across
all content areas and all grade levels. Each requires specific implementation techniques to produce
the effect sizes reported, so their use requires learning to use them correctly.
TASK: Use the ESD STATS Translator to find the missing values.
1.
Comparing, contrasting, classifying, analogies, and metaphors (Identifying similarities and differences). These
processes are connected as each requires students to analyze two or more elements in terms of their similarities and
differences in one or more characteristics. This strategy has the greatest effect size on student learning. Techniques vary by
age level. (Effect Size(ES)=1.61 or
percentile points)
2.
Summarizing and note-taking. To summarize is to fill in missing information and translate information into a
synthesized, brief form. Note-taking is the process of students’ using notes as a work in progress and/or teachers’
preparing notes to guide instruction. (Effect Size(ES)=
or 34 percentile points)
3.
Reinforcing effort and providing recognition/giving praise. Simply teaching many students that added effort will pay
off in terms of achievement actually increases student achievement more than techniques for time management and
comprehension of new material. Praise, when recognizing students for legitimate achievements, is also effective. (Effect
size=0.80 or 29 percentile points)
4.
Homework and practice. These provide students with opportunities to deepen their understanding and skills relative to
presented content. Effectiveness depends on quality and frequency of teacher feedback, among other factors. (Effect
Size(ES)= 0.77 or
percentile points)
5.
Nonlinguistic representation. Knowledge is generally stored in two forms— linguistic form and imagery. Simple yet
powerful non-linguistic instructional techniques such as graphic organizers, pictures and pictographs, concrete
representations, and creating mental images improve learning. (Effect Size(ES)=0.75 or
percentile points)
6.
Cooperative learning. Effective when used right; ineffective when overused. Students still need time to practice skills and
processes independently. (Effect Size(ES)=0.73 or
percentile points)
7.
Setting objectives and providing feedback. Goal setting is the process of establishing direction and purpose. Providing
frequent and specific feedback related to learning objectives is one of the most effective strategies to increase student
achievement. (Effect Size(ES)=0.61 or
percentile points)
8.
Generating and testing hypotheses. Involves students directly in applying knowledge to a specific situation. Deductive
thinking (making a prediction about a future action or event) is more effective than inductive thinking (drawing
conclusions based on information known or presented.) Both are valuable. (Effect Size(ES)=
or
percentile
points)
9.
Activating Prior Knowledge-Cues, questions. These strategies help students retrieve what they already know on a topic.
Cues are straight-forward ways of activating prior knowledge; questions help students to identify missing information;
advanced organizers are organizational frameworks presented. (Effect Size(ES)=
or 22 percentile points)
1
Marzano, R., Gaddy, B., & Dean, C. (2000). What Works in Classroom Instruction. Aurora, CO. Mid-continent Research for Education and Learning.
http://www.mcrel.org/pdf/instruction/5992tg_what_works.pdf
Revised: May 5, 2017
13
REVIEW OF LITERATURE
INTERVENTION
Statistical
Significance Found
Grades
Examined
Topic
Effect Size(ES)
Improvement Index
Percentile
Increase
Standard
Deviation
Coefficient of
Determination
Given the Interventions listed above, which one would you recommend and why?
Given the interventions listed above, which one would you NOT recommend and why?
ARTICLE:
Chambers, b., Slavin, R., Madden, N. (2010). Small-group Computer Assisted Tutoring to improve Reading Outcomes for
Struggling First and Second Graders. Johns Hopkins University, Baltimore, MD. Retrieved Online June 13, 2012 from
http://www.successforall.org/SuccessForAll/media/PDFs/Small-GroupComputer-assisted-Tutoring-ESJ.pdf
Revised: May 5, 2017
14
BONUS DISCUSSION: Scenario
You are working with a school that is implementing RTI. They have established a set of assessments, for each grade level
that the teachers have agreed to use. The school has an RTI team that includes two Title I teachers, an intervention
specialist, a special education teacher, the principal and an education assistant who tutors students in reading
throughout the day. The RTI team facilitates CAST meetings for grade level teachers three times a year. These meetings
are used to place students in tiered groups for reading. The process works like a well-oiled machine. Each teacher brings
a spreadsheet to the meeting that has all their students’ test scores for the year. The team reviews the data and discusses
each student’s next reading placement.
During the spring round of CAST meetings, the teachers realized that they had not changed the shape of the curve for
their students. In each grade level, the year had started with an average of seven students in tier I, the majority of
students were in tier II and an average of four students in tier III. In tier II, some students had shifted positions but there
was no real change in the shape of their student distribution.
The principal noted that the RTI team did not talk about instruction. Instead they focused on test scores and placing
students into reading groups. What happened in those groups was not a part of the conversation. The principal
understood that classroom instruction was a sensitive topic and the teachers were not in the habit of planning together,
discussing their instructional practices or observing each other’s classrooms.
The principal is looking to the data coach to help establish a data-informed conversation about instructional practice. She
believes that data will help keep the conversation more objective. She also wants her teachers to begin planning
instruction together so when they share students, they will know what the students are doing in each of the tiered
classrooms. As a data coach, How might you go about helping the RTI team explore instructional practices from a databased perspective?
Revised: May 5, 2017
15
Data Informed Instructional Inquiry
The process below is designed to support teachers using their student level data to determine an instructional activity
they can test to produce their own local about knowledge about how effective a particular instructional activity is for
accomplishing the hoped for effect.
Use the following protocol to work through the student level data and design a local inquiry.
1. Identify the instructional
concern
2. Determine which students
the inquiry will be focused on
3. Identify the particular
instructional target
Using the effect size calculator determine the size of the effect that might be
necessary to get students in a chosen grade to meet expectations.
Looking at the range of students test scores, the mean, median and the mode
choose one percentile group to focus the inquiry on.
Given the chosen group of students, determine a discrete skill, concept, or
behavior that will be taught.
4. Recalculate the effect size
for the smaller group
Using the effect size calculator determine how large an effect the instruction
will need to have for the target students to reach expectation.
5. Per and Post Assessment
Determine a per and post assessment that can be used to clarify where the
group is starting from and whether the instructional strategy had an effect
Looking at the collection of strategies with effect sizes., choose a strategy that
has proven to likely produce the size of an effect that you want and develop an
implementation plan
6. Instructional practice
7. Give the pre test
8. Implement the instructional
plan
9. Give the post test
Revised: May 5, 2017
16
Assessing Evidence-Based Programs and Practices (Hexagon Tool) - March, 20122
Capacity
 Staff meet min. qualifications
 Able to sustain Impact Drivers
o Financially
o Structurally
 Buy-in process operationalized
o Practitioners
o Families
o Agency
Need in school, district, state



Academic & Socially Significant issues
Parent and Community perceptions of need
Data Indicating Need
Fit with current Initiatives



School, district, state priorities
Organizational structures
Community Values
NEED
CAPACITY TO
IMPLEMENT
INTERVENTION
READINESS
FOR
REPLICATION
Assessing
EvidenceBased
Programs
and
Practices
FIT
RESOURCE
AVAILAILITY
EVIDENCE
Readiness
 Qualified purveyor
 Expert or TA available
 Mature sites to observe
 Several replications
 How well is it operationalized
 Are important Drivers
operationalized?
Resources and Supports for,
Evidence
 Outcome = Is it worth it?
 Fidelity data
 Cost – Effectiveness data
 Number of studies
 Population similarities
 Diverse cultural groups
 Efficacy of effectiveness







Curricula & Classroom
Technology supports
Staffing
Training
Data Systems
Coaching & Supervision
Administration & system
Revised: May 5, 2017
17
Evidence-Based Planning2
NEED
5 Point rating Scale (High = 5 Medium = 3 Low = 1)
Midpoints can be used and scored as a 2 or 4
Low
Medium
High
1
2
3
4
5
FIT
1
2
3
4
5
RESOURCES AVAILABILITY
1
2
3
4
5
EVIDENCE
1
2
3
4
5
READINESS FOR REPLICATION
1
2
3
4
5
CAPACITY TO IMPLEMENT
1
2
3
4
5
2
Assessing Evidence-Based Programs and Practices (Hexagon Tool)- March, 2012- State Implementation and Scaling up Evidence-based Practices (SISEP) Center FPG Child Development
Institute, University of North Carolina, Chapel Hill http://sisep.fpg.unc.edu/resources/assessing-evidence-based-programs-and-practices-hexagon-tool
Revised: May 5, 2017
18
EXCEL CHEAT SHEET
Function
Description
AVERAGE
Returns the arithmetic mean and the specified numbers and takes the form
=AVERAGE(number1,number2,…), where the numbers can be names, arrays, or references that resolve
to numbers. Cells containing text, logical values, or empty cells are ignored, but cells containing a zero
value are included.
COUNTIF
Counts the number of cells within a range that match specified criteria and takes the form
=COUNTIF(range,criteria), where range is the range you want to test and criteria is the logical test to be
performed on each cell.
FREQUENCY
Returns the number of times that values occur within a population and takes the form
=FREQUENCY(data_array,bins_array).
MEDIAN
Computes the median of a set of numbers, takes the form =MEDIAN(number1,number2,…), and can
accept up to 30 arguments, ignoring text, error values, and logical values.
MODE
Determines which value occurs most frequently in a set of numbers, takes the form =MODE(number1,
number2,…), and can accept up to 30 arguments, ignoring text, error values, and logical values.
PERCENTILE
Returns the member of an input range that is at a specified percentile ranking and takes the form
=PERCENTILE(array,k), where array is the input range and k is the rank you want to find.
PERCENTRANK Returns a percential ranking for any member of a data set and takes the form
=PERCENTRANK(array,x,significance), where array specifies the input range; x specifies the value whose
rank you want to obtain; and the optional significance indicates the number of digits of precision you
want. If significance is omitted, results are rounded to three digits (0.xxx or xx.x%).
QUARTILE
Returns the value in an input range that represents a specified quarter-percentile and takes the for
=QUARTILE(array,quart).
RANK
Returns the ranked position of a particular number within a set of numbers and takes the form
=RANK(number,ref,order).
STANDARDIZE Returns a normalized value from a distribution characterized by mean and standard_dev and takes the
form =STANDARDIZE(x,mean,standard_dev), where x is the value you want to normalize; mean is the
arithmetic mean of the distribution; and standard_dev is the standard deviation of the distribution.
STDEV
Estimates standard deviation, assuming that the arguments represent only a sample of the total
population, and takes the form =STDEV(number1,number2,…), accepting up to 30 arguments.
VAR
Computes variance, assuming that the arguments represent only a sample of the total population, and
takes the form =VAR(number1,number2,…), accepting up to 30 arguments.
Revised: May 5, 2017
19
TERMS FOR THE DAY 3
Action Research: A type of research in which educators examine their own practice and evaluate strategies to improve
practice and education outcomes.
Bivariate Correlation: A statistical correlation between two variables.
Central Tendency: A score in a set of scores or a frequency distribution that is typical or representative of all the scores.
Measures of central tendency are the mean, median and mode.
Coefficient of determination: For bivariate correlations, the coefficient of determination is defined as r2, which is
interpreted as the proportion of variation in the scores that is explained by the relationship between the variables. Note:
Correlations indicate statistical, not causal, relationships.
Confidence interval: A range of values that indicates the confidence or probability of observing a particular score or value
in a population, usually expressed as standard deviation units above and below the mean.
Correlation coefficient: A number indicating strength & direction of the statistical association between 2 or more
variables.
Descriptive statistics: Statistics used to describe, organize and summarize data.
Effect size: The degree to which a practice, program or policy has an effect based on research results, measured in
standard deviation units. (Effect size is also referred to as practical significance.) A statistic commonly used to measure
effect size is Cohen’s d, which social scientists interpret as the following: d = .2, small; d = .5 to .8, medium; and d = .8 and
higher, large.
Frequency distribution: The frequency of occurrence of scores in a set. Frequency distributions can be represented in
graphs or tables.
Inferential statistics: Statistics used to make inferences about a population based on the scores obtained from a sample.
Inferential statistics are based on the mathematics of probability theory, ex., t, F and Chi Square.
Intervention: A procedure, technique or strategy that is designed to modify an ongoing process. In research studies, the
intervention also is referred to as a treatment. Most interventions in education are designed to modify directly or
indirectly the student-learning process.
Mean: In general, the average score in a set of scores or frequency distribution, calculated as the sum of the scores
divided by the number of scores.
Median: The middle score in a set of scores or frequency distribution such that 50% of the scores are at or below the
median score.
Mode: The most frequent score in a set of scores or a frequency distribution.
N (n): The number of scores in a population (N) or a sample (n) of scores.
3
Education Commission of the States (ECS) and Mid-continent Research for Education and Learning(McREL) (2004, February) A Policymaker’s Primer on Education Research: How To
Understand, Evaluate and Use. Retrieved online June 13, 2012 http://www.ecs.org/html/educationIssues/Research/primer/understandingtutorial.asp
Revised: May 5, 2017
20
Normal curve: The bell-shaped curve that results from the graph of a normal frequency distribution.
Normal distribution: A symmetrical frequency distribution in which the scores form a bell-shaped curve, and the mean,
median and mode have the same value.
Percent: The proportion of participants who obtain a particular score in a frequency distribution.
Percentile: The percent of participants who score at or below a particular score in a frequency distribution (also referred
to as percentile rank).
Population: All individuals or entities belonging to the group that is being studied.
Practical significance: The degree to which a practice, program or policy has enough of an effect to justify its adoption.
Practical significance usually is measured with statistics that calculate effect sizes.
Range: The difference between the highest and lowest score in a set of scores or frequency distribution.
Raw score: An original score on a test or other measuring instrument prior to any score transformations.
Sample: A subset of individuals or entities from a population.
Standard deviation: A measure of the variability of the scores in a set of scores or a frequency distribution, equivalent to
the average distance of the scores from the mean.
Standard score: A score that transforms an original or raw score into standard deviation units in order to locate the
score’s position within a frequency distribution. Standard scores also are known as z-scores and are calculated as: z = Raw
Score – Mean /Standard Deviation. Sign of the score (plus or minus) indicates whether it is above or below the mean.
Statistically significant: A result that has a low probability (e.g., 5 %) of occurring by chance. Because it is unlikely that a
statistically significant result has occurred by chance, the result is said to reflect non-chance factors in the study, such as
the effects of a treatment.
Statistics: Methods and rules for organizing and interpreting quantitative observations.
References and Resources
Cooper, H. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage
Publications.
Creswell, J.W. (2002). Research design: Qualitative, quantitative and mixed method approaches. Thousand Oaks,
CA: Sage Publications.
Isaac S. & Michael, W. B. (1995). Handbook in research and evaluation (3rd ed.). San Diego: EdITS.
Gravetter, F. J. and Wallnau, L. B. (1988). Statistics for the behavioral sciences (2nd ed.). St. Paul: West Publishing.
McMillan, J. H. (2000). Educational research: Fundamentals for the consumer (3rd ed.). New York: Addison
Wesley Longman.
Shadish, W. R., Cook, T.D. and Campbell, D. T. (2002). Experimental and quasi-experimental designs for causal
inference. Boston: Houghton Mifflin.
Shanahan, T. (2000). "Research synthesis: Making sense of the accumulation of knowledge in reading." In M. L.
Kamil, P.B. Mosenthal, P. D. Pearson, and R. Barr (Eds.), Handbook of reading research, volume III (pp. 209–226).
Mahwah, NJ: Lawrence Erlbaum and Associates
Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies (2nd ed.). Upper Saddle River, NJ:
Prentice Hall.
Revised: May 5, 2017
21