Download Using Data from Learning Tools to Improve Student Success

Academic Analytics: Using Data from
Learning Tools to Improve Student
Success
John P. Campbell
Wednesday, March 5, 2008
8:10-9:00 am
Student Success
Student success is defined as an
intentional experience that leads to a
degree, intellectual and personal
growth, and prepares a student for
life and a career in a dynamic, global
society.
2
Forms of Success/Retention
• Course
• Program
• Institution
3
Academic Analytics
• Mining data from systems that
support teaching and learning to
provide customization, tutoring, or
intervention within the learning
environment
• “Actionable intelligence”
4
Process of Analytics
Refine
Adapted from: Eckerson, W. W. (2006) Performance Dashboards: Measuring, Monitoring, and Managing Your
Business. John Wiley & Sons, Inc. Hoboken, NJ
5
Project Phases Analytics
• Exploration with historical data
• Pilot Project: Phase 1
• Pilot Project: Phase 2
• Pilot Project: Expansion
• Collaborative Tools
6
Exploration of Historical Data
Aptitude
Effort
7
Exploration of Historical Data
•
•
•
•
•
Use of analytics to predict which
students need help
Selection of CMS Users
27,276 unique students
597 courses – over 3,000 sections
75 departments and 9 colleges
8
Predictive Power – Overall Population
• Students needing help: 65.7%
• Students doing well: 87.4%
• Overall: 79.3%
9
Usage is Important
Correct Classification of Students Needing Help
66.00%
64.00%
62.00%
60.00%
58.00%
56.00%
54.00%
52.00%
50.00%
Low
Medium
High
CMS Usage
10
Predictive Results
120.00
Percentage Correct
100.00
80.00
60.00
40.00
20.00
0.00
0.50
0.60
0.70
0.80
0.90
Probability of Needing Help
11
Pilot Project: Phase 1
• Gateway Biology course
• 300 students
• “Interventions” based on the
predictive results
• Goal: to encourage students to use
existing resources
12
Phase One: “The Plan”
• Weeks 1, 2, 3 – email messages to
students
• Week 4 – contact from instructor
• Week 5 – contact from advisor
13
Results
• Highest Risk:
–Most remained “at risk”
–Still unlikely to take advantage of
resources
• Lower Risk:
–Majority were able to leave the “at
risk” status – as long as feedback
continued
–More likely to take advantage of
resources
14
Biology Resource Center Usage
120
100
Number of Visits
80
60
40
20
0
Message Period
Post Message Period
Test Group
Non-Test Group
Total Visits
Grades
30
25
Count of Grade
20
15
10
5
0
A
B
C
D
F
Grade Earned
Test Group
Non-Test Group
I
W
Dropped
Student Responses
• “Really appreciate knowing how I'm doing
before I get too far into the course.”
• “Your message was a "kick in the butt" that
woke me up.”
• “You mean, if I get help, I'll do better, and it
won't be counted against me?”
• “This biology lab is the hardest I've ever
taken, but your message let me know that I
need to get more help. Also, I can see that
this lab is helping me in my biology lecture
course, and in my chemistry lab.”
17
Pilot Project: Phase 2
• Refine the messages – tough love for
high risk students
• Continual feedback
18
Student Response
“I've recently been utilizing the BRC and even my
classmates (not cheating of course, just studying
together) to help me understand what I need to have
for each lab/homework assignment. I feel like I've
been understanding the material better and my
homework grades have been improving. I think I
understand what I need to do to continue this rise, so
hopefully you'll be able to see that I'm well on my
way to better grades. Thank you for your support
though. I'm glad that you've set up so may ways for
us to be successful in this class.”
Next Steps: Expansion
• Biology – Phase 3
• Freshman Engineering –
Engineering, Chemistry, and
Physics courses
• Refining the interventions and
increasing student awareness
20
Analytics and Collaboration Tools
• Potential wealth of “engagement”
data
• Wimba actively evaluating the first
series of steps – which data is likely
to lead to results
21
Social Networking Analysis
• What can we say about learning
networks?
22
Lessons Learned
• Privacy is the eye of the beholder
• Move towards data-driven
decision making is a time
consuming process
• Challenge in creating awareness
• Sustainability
• Obligation of knowing
23
How to Start
• Participate in the discussions
• Focus on staff development
• Inventory possible data sources
• Build relationships for future
collaboration
• Create a pilot project
24
Resources
• Academic Analytics Overview:
http://www.educause.edu/ir/library/pd
f/PUB6101.pdf
• EDUCAUSE Review: Academic
Analytics: A New Tool for a New Era
http://www.educause.edu/apps/er/er
m07/erm0742.asp
25