Download Outcomes of the Majors: On Being Deliberate and Explicit

Assessment in Student Affairs:
Some Thoughts on a Tricky Business
Peter T. Ewell
National Center for Higher Education Management Systems
(NCHEMS)
Student Affairs Assessment Conference
Emory University
October 14, 2013
Logic of This Keynote
 Challenges and Dilemmas of Assessment Practice in
Student Affairs: What Makes This a Tricky Business
 Growing Demand for the Kinds of Proficiencies at the
Center of Student Affairs
 Some (Occasionally Odd) Techniques for Getting at
These Proficiencies
 Using Results of Assessment to Improve
Programming and Practice in Student Affairs
A Prefatory Comment
Two Worlds of Assessment:
 Academic: Faculty Want to Know About that Small
Piece of the Student that is Trying to Learn
[Mathematics, History, etc.]
 Student Affairs Professional: Want to Know About
the Experience and Development of the Whole
Student that is Experiencing an Emotional,
Attitudinal, and Cognitive Process of Transformation
Some Particular Challenges of
Assessment in Student Affairs



The Domains of Interest are Not Easy to
“Measure”
Many of Them are Affected by Dispositions to Act
in Particular Ways Even Though Students “Know”
What they Should Do
Attributions of Cause are Hard to Relate to
Individual or Isolable Programmatic Features or
Elements
The Growing Demand for These Kinds of
Proficiencies
 Employer Complaints About Recent College
Graduates [and What They Want Instead]
 Demands for Integrated Abilities and Practical
Competence
 Reflected in Prominent Expectations Frameworks
Like the Lumina Degree Qualifications Profile (DQP)
and AAC&U’s LEAP Goals
Graduate Attributes: Top Choices for
Employers from 2013 AAC&U Survey
Percent Reporting “Very” and “Somewhat” Important:
 Ethical Judgment and Integrity – 96%
 Comfortable with Diverse Backgrounds – 98%
 Demonstrated Capacity for Professional
Development – 94%
 Interest in Giving Back to Communities Our
Company Serves – 71%
Some Proposed Revisions to DQP 2.0
 Confronts the ethical issues present in prominent
problems in politics, economics, society, scholarship,
or the arts and demonstrates how at least two
different ethical perspectives, influence decision
making with respect to such problems
 Describes how knowledge from different cultural
perspectives would affect interpretations of
prominent problems in politics, society, the arts
and/or global relations
Some AAC&U LEAP Goals
 Ethical Reasoning and Action
 Inter-Cultural Knowledge and Competence
 Teamwork and Problem Solving
 Civic Knowledge and Engagement – Global and
Local
 Foundations and Skills for Lifelong Learning
Some Useful Approaches for Gathering
Evidence About Values and Dispositions
 Survey Items and Item Construction
 Time Diaries and the “Beeper Study”
 Formal Focus Groups and Interviews
 Informal Observation and Imagery
 Checklists
 Rubrics Applied to Behavior and Work Products
 Inventories of Good Practice
Survey Items and Item Construction
 Scales with Even Numbers of Decision Points
 “Semantic Differential” Items
 Scenario Based Items (e.g. The Defining Issues
Test)
 Open-Ended Items (e.g. “Areas and Agents of
Change”)
Semantic Differential Items Example
“Areas and Agents of Change” Example
Observation and Imagery
 Develop and Pilot Clear Observational Protocols
 Ensure What You are Looking at is Typical
 For Ongoing Processes, Observe Regularly at
Defined Points in Time
 Consider Using Photographs or Videos to Capture
Demographic Mixes or Ongoing Patterns of Behavior
Observation and Imagery
 Develop and Pilot Clear Observational Protocols
 Ensure What You are Looking at is Typical
 For Ongoing Processes, Observe Regularly at
Defined Points in Time
 Consider Using Photographs or Videos to Capture
Demographic Mixes or Ongoing Patterns of Behavior
Some Applications of Checklists
 Academic Advising
 Service Learning Courses
 Internships, Field Placements
 Student inquiries
 Academic Skills Centers
 Residence Hall and Student Group Advisors
AAC&U VALUE Rubric for Teamwork
Capstone
4
Milestones
Benchmark
1
3
2
Contributes to Team Meetings
Helps the team move forward by articulating
the merits of alternative ideas or proposals.
Offers alternative solutions or courses of action Offers new suggestions to advance the work of
that build on the ideas of others.
the group.
Shares ideas but does not advance the work of
the group.
Facilitates the Contributions of Team Members
Engages team members in ways that facilitate
their contributions to meetings by both
constructively building upon or synthesizing the
contributions of others as well as noticing when
someone is not participating and inviting them
to engage.
Engages team members in ways that facilitate
their contributions to meetings by constructively
building upon or synthesizing the contributions
of others.
Engages team members in ways that facilitate
their contributions to meetings by restating the
views of other team members and/or asking
questions for clarification.
Engages team members by taking turns and
listening to others without interrupting.
Individual Contributions Outside of Team Meetings
Completes all assigned tasks by deadline;
work accomplished is thorough,
comprehensive, and advances the project.
Proactively helps other team members
complete their assigned tasks to a similar level
of excellence.
Completes all assigned tasks by deadline;
work accomplished is thorough,
comprehensive, and advances the project.
Completes all assigned tasks by deadline;
work accomplished advances the project.
Completes all assigned tasks by deadline.
Fosters Constructive Team Climate
Supports a constructive team climate by doing Supports a constructive team climate by
all of the following:
doing any three of the following:
•
Treats team members respectfully by •
Treats team members respectfully by
being polite and constructive in
being polite and constructive in
communication.
communication.
information,
[email protected]
•
Uses positive vocalforormore
written
tone,
•please contact
Uses positive
vocal or written tone,
facial expressions, and/or body
facial expressions, and/or body
language to convey a positive
language to convey a positive
attitude about the team and its work.
attitude about the team and its work.
•
Motivates teammates by expressing •
Motivates teammates by expressing
confidence about the importance of
confidence about the importance of
the task and the team's ability to
the task and the team's ability to
accomplish it.
accomplish it.
•
Provides assistance and/or
•
Provides assistance and/or
encouragement to team members.
encouragement to team members.
Supports a constructive team climate by
doing any two of the following:
•
Treats team members respectfully by
being polite and constructive in
communication.
•
Uses positive vocal or written tone,
facial expressions, and/or body
language to convey a positive
attitude about the team and its work.
•
Motivates teammates by expressing
confidence about the importance of
the task and the team's ability to
accomplish it.
•
Provides assistance and/or
encouragement to team members.
Supports a constructive team climate by doing
any one of the following:
•
Treats team members respectfully by
being polite and constructive in
communication.
•
Uses positive vocal or written tone,
facial expressions, and/or body
language to convey a positive
attitude about the team and its work.
•
Motivates teammates by expressing
confidence about the importance of
the task and the team's ability to
accomplish it.
•
Provides assistance and/or
encouragement to team members.
Addresses destructive conflict directly and
Identifies and acknowledges conflict and stays
constructively, helping to manage/resolve it in a engaged with it.
way that strengthens overall team
cohesiveness and future effectiveness.
Redirecting focus toward common ground,
toward task at hand (away from conflict).
Passively accepts alternate
viewpoints/ideas/opinions.
TEAMWORK VALUE RUBRIC
Responds to Conflict
Evidence-Based Management: What Does a
“Culture of Evidence” Require?
 Respect for the Facts
 Clear Goals and Objectives
 Concrete Evidence About the Achievement of Goals
and Objectives
 An Attitude Toward Problem-Solving that Avoids
“Finger-Pointing”
 Clear Follow-Through On Decisions and Why They
Were Made
 Willingness to Stop Doing Things When They Don’t
Work
Strategies for Addressing Key Difficulties
in Building a Culture of Evidence
 Keep the Outcomes Framework Simple
 Look at Data Distributions, Not Just “Averages”
 Adjust the Approach to Accommodate Important
Differences in Context
 Share Approaches Across Units
 Don’t Wait for Perfection
Some “Points of Attack” in Building
Cultures of Evidence
 Expectations Exercises
 Use Results to Inform and Initiate Discussions,
Rather than “Give Answers”
 Initiate Visible Opportunities to Consider Results
and What They Mean
 Present Results as Discrepancies
 Disaggregate Results to Show Variations
 Package Results Around Perceived Problems or
Embed Them in Regular Decision Processes
Some Final Reminders
 If It’s Important, Try to Assess It – No Matter How
“Tricky” the Business Seems to Be
 Dispositions Matter – So Experiment with Indirect
and Unobtrusive Approaches
 Use Multiple Forms of Evidence Wherever Possible
 Be Creative and Have Fun