Download ETS Proficiency Profile Analysis

General
Education
Entry
Level or
Developmental
Student
Learning
Program
Level
Institution
Level
• Academic Excellence
•
•
•
•
•
Enrollment Management
Diversity
Resources
Accountability
Community Engagement
Source: Rogers State University Strategic Plan
Formative
Summative
Formative
Summative
Entry
Level or
Developmental
General
Education
Student
Learning
Program
Level
Institution
Level
Formative
Summative
Formative
Summative
• Direct (cognitive)
• Indirect (affective)
• Direct (cognitive)
• Indirect (affective)
• Direct (cognitive)
• Indirect (affective)
• Direct (cognitive)
• Indirect (affective)
1.
2.
3.
4.
Acquire and evaluate
information.
Analyze and integrate
knowledge.
Develop perspectives
and an understanding of
the human experience.
Communicate
effectively.
Refer to Section II of Annual Student Assessment Report for 2011-2012 to OSRHE
OSHRE requirement:
• Evaluate student
progress in general
education
formatively
(between 45 and 60
credit hours)
• Report annually
General
Education
Entry
Level or
Developmen
tal
Student
Learning
Program
Level
Institution
Level
 Are RSU students learning to think
critically, read and write at
appropriate levels?
 “Appropriate levels” is operationally defined as
scores on the ETS Proficiency Profile equivalent
with other bachelor’s degree-conferring
institutions.
 Formative measure occurring between 45 and 60
credit hours (OSHRE definition)
ETS: Educational Testing Service
1.
2.
3.
4.
RSU Gen Ed SLOs
ETS Proficiency Profile
Acquire and evaluate
Factors
information.
1. Reading
Analyze and integrate
2. Writing
knowledge.
3. Mathematics
Develop perspectives
and an understanding of 4. Critical Thinking
the human experience.
Communicate
effectively.
Repeated Measures Design
Fall 2011 and
2012 Entering
Freshmen
Fall 2010 and
2011 Entering
Freshmen
Between 45 – 60
credit hours
Pretest
Learning
Posttest
Learning
Posttest
RSU Fall 2011 &
2012 Entering
Freshmen
Pretest
Peer Fall 2007-12
Entering
Freshmen
Pretest
Between 45 – 60
credit hours
RSU Spring 2012
& 2013
Sophomores
Learning
Posttest
Peer 2007 - 2012
Sophomores
Learning
Posttest


Students with no General Education
Fall 2011 and Fall 2012 first-time freshmen
▪
▪
▪
▪
No transfer hours
No concurrent enrollment hours
No 100% online
No Bartlesville
 Fall 2011: 110 students randomly selected from 472
entering freshmen* (baccalaureate and associate)
▪ N = 80 participants (one not completed) or 73% response rate
 Fall 2012: 94 students identified (baccalaureate only)
▪ N = 69 participants or 73% response rate
▪ 70 students needed for 95% confidence level
 Incentive of $10 on Hillcat Card; enrollment hold
*Based on capacity of Testing Center

Students with General Education at RSU
 Spring 2012 sophomores
▪
▪
▪
▪
Had 45 – 60 credit hours including Spring 2012
Did not transfer to RSU
Did not earn concurrently enrolled credits in high school
Were not 100% online
 Spring 2012: 120 students randomly selected from 141
eligible
▪ 93 needed for 90% confidence level
▪ N = 17 participants or 14% response rate
 Spring 2013: 70 students identified
▪ N = 55 or 79% response rate (60 needed for 95% confidence level)
 Incentive of $10 on Hillcat card; enrollment hold added
spring 2013
*Based on capacity of Testing Center (two additional computers available)

Entering Freshmen Testing
 Campus Testing Centers
 August – September (or until completed)

Sophomore Testing
 Campus Testing Centers
 January – February
(or until March 31)
 Select peers by number and type of institutions in
database
▪ Entering Freshmen: Public Baccalaureate
institutions only (July 2007 – June 2012)
▪ 19 Institutions
▪ 28,741 students (RSUfall2012 n= 69; RSUfall2011 n=79**)
▪ Sophomores: Public Baccalaureate institutions only
▪ 15 Institutions
▪ 27,664 students (RSUspring2013 n = 55; RSUspring2012 n=17)
*Must select at least 10 institutions. **Included associate degree-seeking students
Mean size = 1,513 of institution cohort bank











Athens State University (AL)
Coastal Carolina University (SC)

Colorado State University

Pueblo (CO)
Dickinson State University (ND) 
Elizabeth City State University

(NC)
Lander University (SC)

Lewis-Clark State (ID)
Lock Haven University (PA)
Metropolitan State College of 
Denver (CO)

Missouri Southern State
University (MO)

Missouri Western State
University (MO)
Shawnee State University (OH)
Shepherd University (WV)
University of Maine – Fort Kent
(ME)
University of Maine – Presque
Isle (ME)
University of South Carolina –
Aiken (SC)
University of South Carolina –
Upstate (SC)
West Virginia University Parkersburg (WV)
Winston-Salem State University
(NC)
Mean size = 1,848 of institution cohort bank








Athens State University
(AL)
Coastal Carolina University 
(SC)
Colorado State University 
Pueblo (CO)
Dickinson State University 
(ND)
Elizabeth City State

University (NC)
Lander University (SC)

Metropolitan State College
of Denver (CO)

Missouri Southern State
University (MO)
Missouri Western State
University (MO)
Shawnee State University
(OH)
Shepherd University (WV)
University of South Carolina
– Aiken (SC)
University of South Carolina
– Upstate (SC)
West Virginia University Parkersburg (WV)
Winston-Salem State
University (NC)
Max
500
490
Peer standard deviation = 6.8
RSU standard deviation = 16.6
Fall 2011 Mean = 431.9 included associate-degree seekers
480
470
Peer Mean = 439.7
460
450
438.1
440
430
420
R
Q
P
O
N
M
L
K
J
I
H
G
F
RSU
E
E
D
C
B
Min 400
A
410
Max
Peer Mean
RSU Mean
130
125
120
116.6 117.2
115
113.3 112.7
110.7 110.5
112.2 111.4
110
105
Min
100
Critical Thinking
s = 1.9
Reading
Writing
s = 2.4
s = 1.4
Mathematics
s = 1.7
Max
Peer Mean
RSU Mean
130.0
125.0
120.0
115.0
114.0
114.3
112.5
114.2
114.5
112.0
110.0
105.0
Min
100.0
Humanities
s = 1.9
Social Sciences
s = 1.9
Natural Sciences
s = 1.8
Proficient
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Marginal
Not Proficient
Proficient
Marginal
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
RSU classifications are bright colors ; Peer classifications are darker colors
Not Proficient
Max
Peer standard deviation = 6.8
RSU standard deviation = 17.7
500
490
480
470
Peer Mean = 439.6
460
450
441.5
440
430
420
410
Min 400
A
B
C
D
E
F
G
H
I
RSU J
K
L
M
N
Peer Mean
Max
RSU Mean
130.0
125.0
120.0
116.6
115.0
110.7
117.6
113.3
111.8
114.5
112.0
110.8
110.0
105.0
100.0
Min
Critical Thinking
s = 1.8
Reading
s = 2.4
Writing
s = 1.4
Mathematics
s = 1.7
Peer Mean
Max
RSU Mean
130.0
125.0
120.0
115.0
114.0
115.2
112.6
112.9
114.2
114.8
110.0
105.0
100.0
Min
Humanities
s = 1.9
Social Sciences
s = 1.9
Natural Sciences
s = 1.7
Proficient
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Marginal
Not Proficient
Proficient
Marginal
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
RSU classifications are bright colors ; Peer classifications are darker colors
Not Proficient
Composite Score
450
438.1
439.7
441.5
430
439.6
410
390
370
350
RSU Entering
Freshmen ETS Entering
Freshmen
RSU
Sophomores
*ETS database for Baccalaureate Colleges only
ETS
Sophomores
N
ACT
Composite
RSU Fall 2011 Freshmen
79
19.7
RSU Spring 2012 Sophomores
17
22.1
RSU Fall 2012 Freshmen
69
21.5
RSU Spring 2013 Sophomores
54
21.8
RSU Cohort
Is there a difference in ETS PP
total scores between RSU
freshmen and sophomores when
controlling for ACT?
ETS PP Total Score
450
438.1
441.5
430
410
390
370
350
RSU Entering Freshmen
RSU Sophomores

Likely generalizable samples of Freshmen
and Sophomores
 Trust but verify



RSU students enter as freshmen on par with
normative database
RSU sophomores have greater gains than
normative sample
RSU sophomores experience increases in
general education accounting for ACT


Continue testing with increased sample sizes
Test exiting seniors


Level 1 - Students who are proficient can:
 recognize factual material explicitly presented in a reading
passage
 understand the meaning of particular words or phrases in the
context of a reading passage
Level 2 - Students who are proficient can:
 synthesize material from different sections of a passage
 recognize valid inferences derived from material in the passage
 identify accurate summaries of a passage or of significant sections
of the passage
 understand and interpret figurative language
 discern the main idea, purpose or focus of a passage or a
significant portion of the passage

Level 3/Critical Thinking - Students who are proficient can:
 evaluate competing causal explanations
 evaluate hypotheses for consistency with known facts
 determine the relevance of information for evaluating an
argument or conclusion
 determine whether an artistic interpretation is supported by
evidence contained in a work
 recognize the salient features or themes in a work of art
 evaluate the appropriateness of procedures for investigating a
question of causation
 evaluate data for consistency with known facts, hypotheses or
methods
 recognize flaws and inconsistencies in an argument


Level 1 - Students who are proficient can:
 recognize agreement among basic grammatical elements (e.g.,
nouns, verbs, pronouns and conjunctions)
 recognize appropriate transition words
 recognize incorrect word choice
 order sentences in a paragraph
 order elements in an outline
Level 2 - Students who are proficient can:
 incorporate new material into a passage
 recognize agreement among basic grammatical elements (e.g.,
nouns, verbs, pronouns and conjunctions) when these elements
are complicated by intervening words or phrases
 combine simple clauses into single, more complex combinations

Level 3 - Students who are proficient can:
 discriminate between appropriate and inappropriate use
of parallelism
 discriminate between appropriate and inappropriate use
of idiomatic language
 recognize redundancy
 discriminate between correct and incorrect constructions
 recognize the most effective revision of a sentence

Level 1 - Students who are proficient can:
 solve word problems that would most likely be solved by
arithmetic and do not involve conversion of units or
proportionality. These problems can be multistep if the steps are
repeated rather than embedded
 solve problems involving the informal properties of numbers and
operations, often involving the Number Line, including positive
and negative numbers, whole numbers and fractions (including
conversions of common fractions to percent, such as converting
"1/4" to 25%)
 solve problems requiring a general understanding of square roots
and the squares of numbers
 solve a simple equation or substitute numbers into an algebraic
expression

Level 2 - Students who are proficient can:
 solve arithmetic problems with some complications, such as
complex wording, maximizing or minimizing, and embedded
ratios. These problems include algebra problems that can be
solved by arithmetic (the answer choices are numeric)
 simplify algebraic expressions, perform basic translations, and
draw conclusions from algebraic equations and inequalities.
These tasks are more complicated than solving a simple
equation, though they may be approached arithmetically by
substituting numbers
 interpret a trend represented in a graph, or choose a graph that
reflects a trend
 solve problems involving sets; problems have numeric answer
choices

Level 3 - Students who are proficient can:
 solve word problems that would be unlikely to be solved by arithmetic;





the answer choices are either algebraic expressions or numbers that do
not lend themselves to back-solving
solve problems involving difficult arithmetic concepts, such as
exponents and roots other than squares and square roots, and percent
of increase or decrease
generalize about numbers (e.g., identify the values of (x) for which an
expression increases as (x) increases)
solve problems requiring an understanding of the properties of integers,
rational numbers, etc.
interpret a graph in which the trends are to be expressed algebraically
or one of the following is involved: exponents and roots other than
squares and square roots, percent of increase or decrease
solve problems requiring insight or logical reasoning
Peers N = 27,664
RSU N = 55
ETS mandates >50 respondents to generate a report
 RSU implemented the abbreviated version
▪ Each student receives 1/3 of test
 Peers from Freshman cohort not in Sophomore cohort
 Lewis-Clark State (ID)
 Lock Haven University (PA) graduate
 University of Maine – Fort Kent (ME)
 University of Maine – Presque Isle (ME)



Is there a difference in ETS PP total scores
between RSU freshmen and sophomores when
controlling for ACT?
Ho: M1 = M2
There is no difference in mean ETS PP total scores as a
function of freshman (no gen ed course work) and sophomore
status (three semesters of gen ed course work at RSU) when
controlling for ACT
where M1 = mean ETS PP total score for fall 2012 sample of entering RSU freshmen
where M1 = mean ETS PP total score for spring 2013 sample of RSU sophomores
Alpha < .05; 95% confidence level

Reject the null hypothesis. There is indeed
a difference (increase) in mean ETS PP total
score between freshmen and sophomores
 Even when controlling for the 0.3 greater
sophomore mean ACT score
 F = 8.871; Critical value of F = 3.949

Significant difference at the 99%
confidence level