Download Measurement changes what is being measured: what does this mean

Measurement changes what is being measured:
what does this mean for the measurement of
quality and productivity in Higher Education?
Associate Professor Alan Bain
Faculty of Arts and Education, CSU
Dr Nick Drengenberg, Deputy Director, Office of Governance
Services & Academic Secretary, CSU
Three Goals
• Describe the problem of measuring quality and productivity
• Describe the current response to the problem
• Describe a better solution to the problem
What We Assume, Hope for, Expect…..
Accreditation/
Standards
Practice
Quality
Productivity
Student/University
Outcomes
What We know
Between 2003–04 and 2013–14, prices for undergraduate tuition, fees,
room, and board at public institutions in the United States (US) rose 34
percent after adjustment for inflation (USDOE, 2014). Costs are expected
to continue to grow at 5% per year yielding an estimated figure of 40,935
dollars for a 4-year degree from a public institution and 90,576 dollars for
the same degree from a private institution by 2030 (Badkar, 2014).
A serious Problem
Productivity ?
Quality ?
What We Assume, Hope for, Expect…..
Accreditation
Standards
Practice
Quality
Productivity
Student
University
Outcomes
Do Current Standards Drive Better Practice?
“ Assessment tasks
for the course of
study and its units
provide opportunities
for students to
demonstrate
achievement of the
expected student
learning outcomes
for the course of
study.” (TEQSA,
2015).
vs
Use current clinical
guidelines relevant to
general practice to
assist in the diagnosis
and management of
our patients (RACGP,
2015).
Clinical Guidelines
-
-
Monitor for electrolyte imbalance
Administer electrolytes as required
Maintain equipment for administering and monitoring electrolytes
Consult physician on administration of relevant medications
Administer medications to help retain or excrete electrolytes as prescribed, if appropriate
Monitor for loss of electrolyte-rich fluids from patient
Institute measures to control excessive electrolyte loss, as appropriate
Provide diet appropriate for patient's electrolyte imbalance
Instruct the patient and/or family on specific dietary modifications, as appropriate
Provide a safe environment for the patient with neurological and/or neuromuscular manifestations of
electrolyte imbalance
Promote orientation
Teach patient and family about the type, cause, and treatments for electrolyte imbalance, as
appropriate
Consult physician if signs and symptoms of fluid and/or electrolyte imbalance persist or worsen
Monitor patient's response to prescribed electrolyte therapy
Place on cardiac monitor, as appropriate
Treat cardiac arrhythmias, according to policy
Prepare patient for dialysis, as appropriate
“Assessment tasks for the course of
study and its units provide
opportunities for students to
demonstrate achievement of the
expected student learning outcomes
for the course of study.”
What We Assume, Hope for, Expect…..
Accreditation
Standards
Practice
Quality
Productivity
Student
University
Outcomes
Is there professional practice at scale in Higher Education?
Comparable
Visible
Workable
Distinctions
Professionally
Controlled
Quality
Practice
Scale
( Bowker & Starr, 2000; Drengenberg & Bain, 2016)
"The lack of acceptable standards for
pedagogy and classroom methodology is
the key reason for the difficulties applying
the concept of educational malpractice to
ineffective teaching in litigation in the
United States, the United Kingdom, and
Australia."
(Mawdsley & Cumming, 2008).
In describing what was effective teaching
the prevailing view was that good teaching
was directly related to having a doctoral
qualification in their research discipline….
“all you need to be a good teacher is to
have your doctorate”
(Scott & Scott, 2015, p.8).
What We Assume, Hope for, Expect…..
Accreditation/
Standards
Practice
Quality
Productivity
Student/Univer
sity Outcomes
Can we determine Quality?
“Educational quality is the
‘elephant in the room’ in
most discussions of
higher education
productivity.”
QILT
“The panel looked carefully at the
prospects for developing the kinds of
comprehensive learning quality
measures needed…We would have
liked nothing better than to propose
such measures but, unfortunately, we
were forced to conclude that this will
not be possible anytime soon.”
Massy et al, Improving Measurement of Productivity in Higher Education,
National Research Council Report, 2012.
What We Assume, Hope for, Expect…..
Accreditation/
Standards
Practice
Quality
Productivity
Student
University
Outcomes
Student Outcomes
Most of the variance in student performance is predicted by
their characteristics at entry. The gains in performance from
entry to graduation are small and there is much more
variability in student performance within universities than
between them indicating the lack of variance contributed by
a university’s approach.
(ACT, 2009; Liu, 2008; Pascarelli & Trenzini, 2005).
Impersonation?
Consequences
(Malpractice)
Reward
Workable
Distinctions
Valid
Standards
Recognition
Quality
Professionally
Controlled
Practice
At Scale
Accredited
Preparation
and Training
Bowker & Starr, 2000; Bain & Drengenberg, 2016; Drengenberg & Bain, 2016; Bain & Zundans-Fraser, (in progress)
What are we Doing?
Measure what we have rather than what we need
The 3 R’s
Rankings, Research, Real Estate
(Craig, 2015)
Efficacy and
quality
measured by
often unrelated
inputs?
Is this gap
real?
https://www.jisc.ac.uk/guides/activity-data-delivering-benefits-from-the-data-deluge
Analytics?
Consequences
(Malpractice)
Reward
Workable
Distinctions
Valid
Standards
Recognition
GAP
Accredited
Preparation
and Training
Is the gap the missing professional framework?
Measuring Productivity without Quality?
Activity
Resources
Completions
Massy, (2016) Re-engineering the University
Time
Meeting standards by getting Busy with Connoisseurship
Not surprisingly, after two or more decades of international study there is
no body of evidence showing a discernable effect of quality assurance
approaches on learning and teaching in higher education.
(Ewell, 2010; Harvey & Newton, 2007; Stensaker, 2008).
https://www.youtube.com/w
atch?v=rmFD3ycPAzY
RESULT: SAMENESS
The lack of enduring and unique identities in higher education
offers an opportunity for education leaders, as it indicates
there are a host of undifferentiated brands, ripe for disruption.
(Dvorak & Busteed, p.2).
Productivity cannot be measured
without an understanding of Quality
Quality cannot be determined
without an understanding of
ProfessionallyControlled Practice at
Scale
Consequences
(Malpractice)
Reward
Workable
Distinctions
Valid
Standards
Recognition
GAP
Accredited
Preparation
and Training
How do we bridge the gap?
Building a Professional Framework
Assertive
Standards
Professional
-ization of
Governance
Valid Data
Professional
-ization of
Learning
and
Teaching
Enabling
Technology
What We Assume, Hope for, Expect…..
Accreditation
Standards
Practice
Quality
Productivity
Student
University
Outcomes
Assertive Standards
Teaching and
learning activities are
arranged to foster
progressive and
coherent
achievement of
expected learning
outcomes throughout
each course of study.
VS
Teaching and
learning activities are
evidence-based and
demonstrate their
effect on learning
outcomes throughout
each course of study.
What We Assume, Hope for, Expect…..
Accreditation/
Standards
Practice
Quality/Produ
ctivity
Student/University
Outcomes
Active Lecture/Online Design
Worked
Examples
Guided
Practice
Independent
Practice
Effects
Feedback .71
Worked Examples .57
Characteristics .65
Haystead & Marzano, 2009; Hattie, 2009; Hattie, 2015
Enabling Technologies
Valid Evidence-Data
Data
need to be
navigational
Governance?
What We Assume, Hope for, Expect…..
Accreditation
/ Standards
Practice
Quality
Productivity
Student
University
Outcomes
Measuring Productivity with Quality
Design
Delivery
Learning
Outcomes
Bain & Zundans Self-Organizing University (2017)
Professional Practice
Quality/Productivity
Valid Standards
Governance
Student
Outcomes
Enabling
Technologies