Download tmplate

Session for NSSE “Veterans”
Regional NSSE User’s Workshop
October 2005
Shimon Sarraf, Research Analyst
Center for Postsecondary
Research, Indiana University
Bloomington
Overview
“NESSIE”
 A Quick Survey
 Reports Reviewed
 Benchmarks Explained
 2005 NSSE Dataset Details
 Future NSSE Developments
 Discussion & Questions
Goals
 Two-way conversation
 Share your experiences
 Good preparation for Shimon’s second and third
session
A Quick Survey to Get Things
Started….
 Years of Participation in NSSE
 Background of attendees
The NSSE Reports:
Respondent Characteristics
 A quick snapshot of
your institution
 Data quality: Confirming
if sample is
representative
 Response Rate and
Sample Error
2005 NSSE Response Rates by
Carnegie Classification
100
90
80
79
77
72
Response Rate (%)
70
58
60
53
49
50
39
40
36
33
32
30
20
17
16
14
10
7
3
0
Doc-Extensive
Doc-Intensive
Masters
Mean
Min
Bac-Gen
Max
Bac-LA
The NSSE Reports:
Respondent Characteristics
What is Sampling Error?
 Assumes random sampling
 An estimate of the margin likely to contain
your "true" score, for example:
 If 60% of your students reply "very often"
and the sampling error is ± 5%, it is likely
that the true value is between 55% and
65%.
 More respondents --> smaller sampling error
2005 NSSE Sampling Error by
Carnegie Classification
30
26
25
Sampling Error (%)
20
15
14
11
10
9
8
6
5
5
4
5
4
2
2
2
2
2
Doc-Extensive
Doc-Intensive
Masters
Bac-Gen
Bac-LA
0
Mean
Min
Max
The NSSE Reports:
Means Comparison
NSSE 2005 Means Comparison Report
NSSEville State University
NSSEville
Variable
Means,
statistical
significance
and effect
sizes
1.
Benchmark
b.
Asked questions in class or contributed to class
discussions
Made a class presentation
Prepared two or more drafts of a paper or
c.
assignment before turning it in
Worked on a paper or project that required
d. integrating ideas or information from
various sources
Included diverse perspectives (different races,
e. religions, genders, political beliefs, etc.) in class
discussions or writing assignments
Come to class without completing readings or
f.
assignments
Worked with other students on projects during
g.
class
Class
CLQUEST
CLPRESEN
ACL
ACL
REWROPAP
INTEGRAT
DIVCLASS
CLUNPREP
CLASSGRP
OCCGRP
Put together ideas or concepts from different
i. courses when completing assignments or during
class discussions
INTIDEAS
Participated in a community-based project (e.g.
k.
service learning) as part of a regular course
Mean
Mean
Sig
a
Master's
Size
b
NSSE 2005
Effect
Mean
Sig
a
Size
b
Effect
Mean
Sig
a
Size
b
In your experience at your institution during the current school year, about how often have you done each of
the following? 1=never, 2=sometimes, 3=often, 4=very often
Worked with classmates outside of class to
h.
prepare class assignments
Tutored or taught other students
j.
(paid or voluntary)
Selected Peers
Effect
Academic and Intellectual Experiences
a.
NSSEville compared with:
TUTOR
COMMPROJ
ACL
ACL
ACL
ACL
FY
2.71
2.55
***
.19
2.84
***
-.15
2.86
***
-.18
***
-.18
SR
3.02
2.82
***
.22
3.15
***
-.16
3.16
FY
2.24
1.95
***
.40
2.30
**
-.07
2.28
*
.11
SR
2.66
2.57
2.93
***
-.32
2.88
***
-.26
FY
2.45
2.48
2.69
***
-.25
2.65
***
-.21
SR
2.32
2.32
2.55
***
-.24
2.51
***
-.20
FY
2.98
2.86
***
.15
3.08
***
-.13
3.08
***
-.12
SR
3.31
3.19
***
.15
3.36
3.37
*
-.08
FY
2.75
2.65
***
.11
2.76
SR
2.74
2.64
*
.11
2.83
**
-.11
2.83
**
-.11
FY
2.14
2.19
2.01
***
.17
2.03
***
.15
SR
2.31
2.28
2.05
***
.35
2.08
***
.31
FY
2.36
2.30
2.42
**
-.08
2.40
2.58
***
-.16
2.52
*
-.09
SR
2.43
2.37
FY
2.41
2.34
SR
2.82
2.80
FY
2.60
2.49
*
.07
*
.08
***
.14
2.77
2.39
2.43
2.76
2.77
2.54
*
.08
2.57
SR
2.89
2.88
2.91
2.93
FY
1.72
1.72
1.68
1.72
SR
1.86
1.87
1.90
1.94
*
-.08
FY
1.39
1.40
1.56
***
-.21
1.54
***
-.19
SR
1.63
1.54
1.80
***
-.19
1.77
***
-.15
*
.11
The NSSE Reports:
Means Comparison
What is Statistical Significance?
 Helps you answer the question, “How likely is
it that the difference between my average
student and the average student at
[comparison group] is due to chance?
 Significance determined by standard alpha
values of p<.05, .01, or .001
The NSSE Reports:
Means Comparison
 Potential problem:
As N becomes large, almost everything
becomes statistically significant
 How do we identify truly significant
differences?
 This is a question of …
practical significance
The NSSE Reports:
Means Comparison
What is Effect Size?
 Practical significance of the mean difference
 ES=mean difference/standard deviation
 .2 is often considered small,
.5 moderate, and
.8 large (but rare!)
 For example, while the difference in the
means is statistically significant, the
difference is so nominal that it doesn’t
warrant further attention
The NSSE Reports:
Detailed Statistics
NSSE 2005 Detailed Statistics
NSSEville State University
First-Year Students
Standard Error of the Mean a
Selected Peers
Master's
NSSE 2005
NSSEville
Selected Peers
Master's
NSSE 2005
NSSEville
Selected Peers
Master's
NSSE 2005
Selected
Peers
Master's
NSSE 2005
Selected
Peers
Master's
NSSE 2005
NSSEville
compared with:
NSSEville
Effect size d
NSSEville
compared with:
NSSE 2005
Significance c
Master's
Number of respondents
Selected Peers
Mean,
N,
SEM,
SD,
p-value,
effect size
Standard deviation b
NSSEville
Mean
CLQUEST
2.71
2.55
2.84
2.86
.02
.02
.01
.00
.82
.82
.84
.84
1,329
3,004
19,955
47,746
.000
.000
.000
.19
-.15
-.18
CLPRESEN
2.24
1.95
2.30
2.28
.02
.01
.01
.00
.77
.74
.78
.78
1,327
3,001
19,949
47,729
.000
.009
.133
.40
-.07
-.04
REWROPAP
2.45
2.48
2.69
2.65
.03
.02
.01
.00
.97
1.01
.97
.98
1,324
2,999
19,939
47,688
.371
.000
.000
-.03
-.25
-.21
INTEGRAT
2.98
2.86
3.08
3.08
.02
.02
.01
.00
.79
.84
.77
.78
1,324
2,999
19,933
47,691
.000
.000
.000
.15
-.13
-.12
DIVCLASS
2.75
2.65
2.76
2.77
.02
.02
.01
.00
.85
.89
.86
.87
1,326
2,998
19,932
47,678
.001
.841
.324
.11
-.01
-.03
CLUNPREP
2.14
2.19
2.01
2.03
.02
.01
.01
.00
.76
.79
.74
.74
1,324
2,994
19,911
47,644
.053
.000
.000
-.06
.17
.15
CLASSGRP
2.36
2.30
2.42
2.40
.02
.02
.01
.00
.81
.84
.81
.82
1,327
3,001
19,948
47,705
.031
.006
.071
.07
-.08
-.05
OCCGRP
2.41
2.34
2.39
2.43
.02
.02
.01
.00
.81
.86
.85
.85
1,327
3,002
19,948
47,718
.012
.304
.428
.08
.03
-.02
INTIDEAS
2.60
2.49
2.54
2.57
.02
.01
.01
.00
.79
.79
.79
.80
1,232
2,835
19,150
45,820
.000
.010
.191
.14
.08
.04
TUTOR
1.72
1.72
1.68
1.72
.02
.02
.01
.00
.82
.84
.83
.84
1,232
2,835
19,160
45,844
.991
.067
.787
.00
.05
.01
COMMPROJ
1.39
1.40
1.56
1.54
.02
.01
.01
.00
.73
.72
.82
.81
1,232
2,832
19,156
45,828
.573
.000
.000
-.02
-.21
-.19
ITACADEM
2.73
2.72
2.57
2.61
.03
.02
.01
.00
1.01
1.01
1.05
1.05
1,233
2,834
19,160
45,850
.846
.000
.000
.01
.15
.12
EMAIL
3.16
2.99
3.01
3.06
.02
.02
.01
.00
.80
.84
.85
.85
1,230
2,833
19,155
45,839
.000
.000
.000
.19
.17
.12
FACGRADE
2.54
2.43
2.62
2.62
.02
.02
.01
.00
.88
.87
.86
.86
1,232
2,835
19,158
45,834
.000
.004
.001
.13
-.09
-.09
FACPLANS
2.25
1.99
2.18
2.16
.02
.02
.01
.00
.85
.86
.89
.89
1,232
2,836
19,159
45,841
.000
.009
.001
.31
.08
.10
FACIDEAS
1.78
1.70
1.82
1.86
.02
.02
.01
.00
.82
.81
.85
.86
1,233
2,833
19,159
45,829
.004
.114
.003
.10
-.05
-.09
FACFEED
2.69
2.60
2.73
2.76
.02
.01
.01
.00
.75
.78
.80
.80
1,212
2,792
18,968
45,387
.000
.047
.001
.12
-.05
-.09
WORKHARD
2.55
2.46
2.64
2.63
.02
.02
.01
.00
.83
.86
.83
.84
1,210
2,795
18,964
45,391
.005
.000
.000
.10
-.11
-.10
FACOTHER
1.48
1.44
1.62
1.63
.02
.01
.01
.00
.76
.73
.84
.84
1,209
2,789
18,965
45,381
.150
.000
.000
.05
-.16
-.18
OOCIDEAS
2.62
2.68
2.69
2.73
.02
.02
.01
.00
.85
.87
.86
.86
1,211
2,792
18,960
45,380
.034
.006
.000
-.07
-.08
-.13
DIVRSTUD
2.52
2.71
2.52
2.60
.03
.02
.01
.00
1.03
1.00
1.01
1.02
1,212
2,790
18,956
45,366
.000
.969
.008
-.19
.00
-.08
DIFFSTU2
2.76
2.85
2.70
2.77
.03
.02
.01
.00
.96
.94
.97
.97
1,212
2,790
18,959
45,375
.009
.035
.726
-.09
.06
-.01
MEMORIZE
2.96
2.88
2.90
2.85
.02
.02
.01
.00
.83
.86
.87
.88
1,207
2,781
18,859
45,096
.013
.016
.000
.08
.07
.12
ANALYZE
3.12
3.05
3.04
3.09
.02
.01
.01
.00
.74
.76
.79
.78
1,207
2,781
18,849
45,068
.004
.000
.113
.10
.11
.05
SYNTHESZ
2.82
2.81
2.81
2.87
.02
.02
.01
.00
.80
.84
.84
.84
1,206
2,779
18,835
45,049
.714
.628
.065
.01
.01
-.05
EVALUATE
2.78
2.74
2.82
2.84
.02
.02
.01
.00
.83
.86
.85
.85
1,207
2,783
18,838
45,064
.136
.100
.020
.05
-.05
-.07
APPLYING
2.99
2.95
2.96
2.99
.02
.02
.01
.00
.81
.86
.85
.85
1,206
2,783
18,847
45,074
.198
.172
.873
.04
.04
.00
READASGN
3.47
3.32
3.23
3.31
.03
.02
.01
.00
.97
.89
.96
.97
1,197
2,771
18,754
44,895
.000
.000
.000
.16
.25
.16
The NSSE Reports:
Detailed Statistics
What are Confidence Intervals?
 CI = Mean +/- 2SEM
 Multiplying the SEM by 2 creates a margin
around the sample mean that is 95% likely
to contain the true population mean.
 More respondents smaller standard error
of the mean (SEM), more precise estimate
 Higher standard deviation greater SEM,
less precise estimate
The NSSE Reports:
Frequency Distributions
Counts and
percentages
for each
response
option
The NSSE Reports:
Frequency Distributions
Tip: Consider merging response options to
create dichotomous variables (1/0)
 Frequently = often + very often
 Substantial = quite a bit + very much
The NSSE Reports: New Features
 Selected Peer Group
 Including or Excluding your targeted oversample
5 Benchmarks of Effective
Educational Practice
Level of Academic Challenge
Active and Collaborative Learning
Student Faculty Interaction
Enriching Educational Experiences
Supportive Campus Environment
Level of Academic Challenge
 Challenging intellectual and creative work is central
to student learning and collegiate quality.
 Institutions promote high levels of achievement by
setting high expectations for student performance.
 11 items include:
 Preparing for class
 Reading and writing
 Using higher-order thinking skills
 Institutional environment emphasizes academic work
Active and Collaborative Learning
 Students learn more when they are more intensely
involved in their education.
 Collaborating with others prepares students to
handle practical, real-world problems.
 7 items include:
 Asking questions in class
 Making presentations
 Working with other students on projects
 Discussing ideas from readings or classes with others
Student Interactions with Faculty
 Interacting with faculty show students first-hand
how to think about and solve practical problems.
 Teachers become role models and mentors for
learning.
 6 items include:
 Discussing assignments with a professor
 Talking about career plans with faculty member or
advisor
 Getting prompt feedback on academic performance
 Working with a faculty member on a research project
Enriching Educational Experiences
 Students need learning opportunities that
complement the goals of the academic program.
 Provide opportunities to integrate and apply
knowledge.
 11 items include:
 Experiencing diversity
 Using technology
 Participating in internships
 Culminating senior experience
Supportive Campus Environment
 Students perform better and are more satisfied at
colleges that are committed to their success.
 Does institution cultivate positive working and social
relationships among different groups on campus?
 6 items include:
 Helping students achieve academically
 Helping students cope with non-academic
responsibilities
 Quality of relationship between student and peers,
faculty, and administrative personnel
Benchmarks of Effective
Educational Practice
How are benchmark scores calculated?
1. Items are converted to a 100-point scale:
[(response value – 1)/(total # of response values – 1)]*100
2. Part-time students' scores are adjusted on four
Academic Challenge items.
3. Student-level scores are created for each group of
items by taking the mean, as long as 3/5ths of the
items were answered.
4. Institutional benchmarks are the weighted averages
of the student-level scores.
Benchmark Report
Level of Academic Challenge
Level of Academic Challenge Items:
75
Preparing for class (studying, reading, writing, rehearsing,
etc. related to academic program)
Number of assigned textbooks, books, or book-length
packs of course readings
65
Benchmark Scores
Challenging
intellectual and
creative work is
central to student
learning and
collegiate quality.
Colleges and
universities
promote high
levels of student
achievement by
emphasizing the
importance of
academic effort
and setting high
expectations for
student
performance
Number of written papers or reports of 20 pages or more;
number of written papers or reports of between 5 and 19
pages; and number of written papers or reports of fewer
than 5 pages
55
Coursework emphasizing analysis of the basic elements of
an idea, experience or theory
45
Nesseville
Coursework
emphasizing
synthesis and organizing of ideas,
Consortium
Carnegie
National
information, or experiences into new, more complex
interpretations and relationships
35First-Year
52.5
52.4
25Senior
56.3
55.6
First-Year
Senior
Nesseville
52.5
56.3
Consortium
52.4
55.6
Carnegie
51.8
54.9
National
53.4
57.0
51.8
53.4
Coursework emphasizing the making of judgments about
the value of information, arguments, or methods
57.0
Coursework54.9
emphasizing application
of theories or
concepts to practical problems or in new situations
Working harder than you thought you could to meet an
instructor's standards or expectations
Campus environment emphasizing time studying and on
academic work
Benchmarks of Effective
Educational Practice
Benchmark recalculation reports (04 & 05):
 Driven by new calculation process that
began for the 04 administration
 Multi-year Comparisons
 EEE: not comparable because of response set
change
 SFI: comparable by removing “research” item
Benchmarks of Effective
Educational Practice
Institutions can use the student-level scores to:
 Investigate what groups are more engaged than
others on your campus.
 Institutional subgroups (i.e., programs, departments)
 Student sub-groups (i.e., gender, race)
 Incorporate scale scores into predictive models of
student outcomes (retention, g.p.a., satisfaction)
Measurement Scales
Satisfaction
General Satisfaction
Satisfaction plus Quality of Campus
Relationships
Campus Environment
Environmental Emphases
Quality of Campus
Relationships
Gains Factors
Personal/Social
General Education
Practical Competence
Deep Learning Activities
 Higher-Order Learning
activities that require students to utilize higher levels of
mental activity than those required for rote
memorization (2b,c,d,e)
 Integrative Learning
activities that require integrating acquired knowledge,
skills, and competencies into a meaningful whole
(1d,e,i,p,t)
 Reflective Learning
activities that ask students to explore their experiences
of learning to better understand how they learn
NSSE Scalelets
 Course Challenge
 Varied Experiences
 Writing
 Information Technology
 Active Learning
 Diversity
 Collaborative Learning
 Support for Student
Success
 Course Interaction
 Out-of-Class Interaction
 Gains (academic,
personal, vocational)
 Interpersonal
Environment
2005 NSSE Dataset Details
What do you need to know to match your
Institutional Report numbers?
 “smpl05” (sampling type)
 use 1 (base sample), 2 (standard oversample), 3
(requested oversample) values…
 and 4 (targeted oversample) if “targetos” equals 1
 “inelig”
 exclude all ineligible respondents
 use those with values of 1 for “inelig”
Future Developments
 Customized Report Engine
 Archiving of reports and datasets online
 Integrating new Carnegie classification
 NSSE Knowledge Base
Discussion and Comments
Shimon Sarraf
Research Analyst
Indiana University Center for Postsecondary Research
1900 East 10th Street
Eigenmann Hall, Suite 419
Bloomington, IN 47406
Ph: 812-856-2169
[email protected]
www.nsse.iub.edu