Download Evaluating Collaboration - University of Wisconsin

Evaluating Collaboration
National Extension Family Life
Specialists Conference
April 28, 2005
Ellen Taylor-Powell, Ph.D.
Evaluation Specialist
University of Wisconsin-Extension
Types of evaluation questions





The outcomes are broad and complex. How do
we get started?
Is evaluating process good enough or do we
have to evaluate outcomes?
Who should be involved in evaluating a
collaborative program?
I’m not in charge. How do I evaluate it?
How do I take credit for something that we’ve
done together?
Issues and challenges

Power – control
Process of the evaluation
 Data






Standards and quality of the evaluation
Cross-cultural issues
Measurements issues
Attribution
Taking credit
Collaborative evaluation
(not evaluation of collaboration)
Since mid – 1970’s, new paradigm of participatory
evaluation
“applied social research that involves trained evaluation
personnel…and practice-based decision makers
working in partnership” (Cousins and Earl, 1992)
 Multiple approaches -from broadening decision making
(practical) to emancipation and social change
(transformation)
 Emphasis on using data collection and feedback to
strengthen and monitor collaboration and thus increase
overall effectiveness and efficiency


value in the process of evaluation, process use (Patton, 1997),
as much as product
Who controls? Who participates? How much?
Researcher control
Consultation
All legitimate
groups
Primary users
Deep participation
Practitioner/participant control
Adapted from Cousins and Whitmore, 1998
First…



Who wants to know what?
For what purpose?
How will information be used?
Building a logic model of collaboration
SITUATION
INPUTS
OUTPUTS
OUTCOMES
Collaborative Product
Collaborative Relationship
Assumptions
External factors
EVALUATION
Collaborative
Effectiveness
Collaboration: Theory of change
Change in
Knowldge
Partners
Attitudes
Implement
activities –
action plan
• Clientele
Skills
•Users
Motivation
holders
Monitor and
evaluate
•Policy
makers
Intent
Researchbased
Communicate
•Publics
Selfefficacy
Funding
Key stake
Advocacy/
Policy
Capacity
building TA
Change in
behaviors
Collaborative
Relationship
building
• Individual
members
• Group
Change in
• KAS
• Selfefficacy
• Intent
Policy
changes
System
changes
Community
changes
Change in
• behaviors
•decision
making
WHAT DO YOU WANT TO KNOW?
Changes in
conditions
Valueadded
•Effective
functioning
partnership
•Member
satisfaction
Evaluating the Collaborative Relationship
1. Process evaluation





How is it functioning? How effective is the group work?
Are we likely to achieve our desired results?
How satisfied are members?
Questions about capacities, operations, climate, context
Factors influencing success
Projected tasks/activities relative to stages of
development
Milestones and Critical Events (journey)
MILESTONES
 Significant points along the way
 Examples
 Key stakeholders on board
 Vision statement established
 Grant secured
 Action plan formulated –
plan of work
 Project
implemented/service
provided
 Project evaluated
CRITICAL EVENTS
 Unexpected events, positive
and negative
 Progress markers
 Evidence of
accomplishments
 Disruptions or obstacles
 Examples
 Change in membership
 Policy change
 New donor added
2. Outcomes (Process outcomes):
 What difference has being a part of this group
made for the individual?



Knowledge, skills, motivations, behaviors, etc.
Human capital development
What difference is their for the group?

Group functioning, identify, resource pooling, etc
Note: Outcomes can be positive, negative or
neutral
Methods









Informal feedback
WHEN?
Member (partner) Survey
Periodic Review
Member (partner) interviews
Points of particular concern
Group discussions
Key informant interviews
Observation
Identification and use of indicators
Network analysis ; sociogram
Use existing materials (integrate into ongoing operations)



Minutes of meetings
Logs: telephone, event, registration forms
Management charts
Tools - Techniques








Community Group Member Survey
Collaborative Relationship scales
Internal collaborative functioning scales
Plan Quality Index
Meeting effectiveness inventories
Stage of readiness
On-line Wilder Collaboration Factors Inventory
(Amherst H. Wilder Foundation)
On-line Partnership self-assessment tool (Center for
Advancement of Collaborative Strategies in Health)
Evaluating Programs/Products
created/implemented by the collaboration
1.

Process or implementation evaluation
(Focus: program delivery vs. coordination or
support role)
How is program being implemented? Fidelity
to plan? Extent of delivery? Participation?
What is/has happened that wasn’t planned?

Outcome evaluation
What is different? For whom? How? To what
extent?
 For: Individuals, Groups/Families, Agencies,
Systems, Communities
 Changes in …

Change in :
Individuals
Attitudes, perceptions, knowledge, competence,
skills, abilities, behaviors, actions, lifestyles
Groups/families Interactions, behaviors, actions, values, culture
Agency,
organization
#/type of services/programs delivered, access,
practices, resource generation, resource use,
policies
Systems
Relationships, interaction patterns, linkages,
networks, practices, policies, resource use,
institutionalization of changes
Communities
Values, attitudes, relations, support systems, civic
action, social norms, policies, laws, practices,
conditions
Tools - Techniques

Monitor implementation
Logs, management charts,
 Interviews
 Observations


Achievement of outcomes
Clientele surveys
 Clientele interviews
 Observations


Mixed Methods
Evaluating self - Taking credit




Mutual (reciprocal) accountability
How do I take credit for my part? How does
Extension gain visibility, recognition?
What is your contribution? What role did you play?
What value did you bring?
Document role you play, your activities and
contributions, inputs you bring, resources you make
available, niche, value…
Your contribution
 Log of activities, roles played
 Record inputs, resources contributed
 Management chart; analysis of minutes
 Independent assessment
Survey
 Interviews


Your (partner) performance:
Most important indicator: other partners’
satisfaction with your performance (Brinkerhoff,
2002)
 Mutual assessment among partners of each
partner’s performance. Resulting discussion re.
Discrepancies = powerful information sharing and
trust building.
(We aren’t very good at this type of thing)

Web address

http://www.uwex.edu/ces/pdande