Download Assessment Rubric for Global Competency in

Proceedings of the 2012 ASEE PSW Conference
Assessment Rubric for Global Competency in Engineering Education
Dianne J. DeTurris*
Cal Poly State University, San Luis Obispo, CA
Abstract
Educating engineering students for global competence is increasingly required to keep up with
the contemporary global environment. As more engineering programs are incorporating global
competency into their curriculum, more attention needs to be paid to how you assess that
competency. Implementation and assessment of international experiences for engineers has been
studied in the last ten years. Largely absent, however, are studies featuring rigorous methods for
assessing competencies specifically related to professional practice within the academic
discipline. Discipline specific measures for assessing international engagement in engineering
need wider implementation.
Only one of the ABET EC2000 Criteria 3 student outcomes mentions the word „global”
explicitly, criteria h. However, the three others that are compatible with assessing global
competency are c, j and k. Of these four criteria, two of these are hard, technical skills, and two
are competencies related to professional practice within the academic discipline. A number of
studies have produced assessment rubrics for measuring global competence in general, but not as
they relate to ABET student outcomes. A rubric is proposed that encompasses both the technical
and the professional skills needed to assess global competency as related to four Criteria 3
outcomes. The rubric includes a spectrum for attitudes, knowledge and skills, an examination of
an internal frame of reference and behavioral observation. Skills related to global competence are
categorized in terms of awareness, perspectives and participation for each of the four student
outcomes with an expanding scale of capability.
*
Director of Global Technical Education Initiatives, Professor of Aerospace Engineering, ASEE Member
DeTurris
Introduction
Change is happening worldwide at a pace so fast that it can be considered revolutionary. Seven
of these areas are defined as the 7 Revolutions. The issues that are most global include
population, resources, technology, communication, economics, conflict and governance1. It is
difficult to keep up with these interconnected challenges. Charles Darwin is credited with once
having said “It is not the strongest of the species that survives, nor the most intelligent, but rather
the most responsive to change”.
In 2008, the NSF funded a National Summit Meeting on the Globalization of Engineering
Education. The outcome of the meeting was a report that describes how technical and
geopolitical developments in the past 20 years have impacted the way engineering is conducted.
The report includes a discussion of the rationale, urgency, the obstacles and hurdles, and the
impact of the current economic downturn, and culminates in the Newport Declaration to
Globalize U.S. Engineering Education2. The declaration presents a compelling case that it is
imperative that all engineering students in the U.S. develop the skills and attitudes necessary to
interact successfully with people from other cultural and national environments. As more U.S.
engineering programs are now incorporating global competency into engineering it becomes
more important to have choices of assessment methods for those outcomes.
Given the imperative stated by the Newport Declaration and the global environment that
engineers are operating in, more definition is needed concerning assessment of the global
component within engineering disciplines. The best way to promote assessment methods is to
reference specific criteria as outlined by ABET (Accreditation Board for Engineering and
Technology). Although EC2000 ABET Criterion 3, Student Outcomes (a-k) only mention the
word „global‟ one time, a global theme can be implemented in a number of other student
outcomes. For example, j is knowledge of contemporary issues. The measureable abilities should
include a component of being knowledgeable about how global issues affect our engineering. A
contemporary issue that is global in nature and affects how we do engineering is the case of how
terrorism affects airline safety.
The engineering community is adding a global component to their programs in individual ways.
For example, Lohmann3 talks about Georgia Tech and defines the general requirements for their
international plan as having three components. The components of global competence education
are
1) Course requirements
2) Second language requirements
3) International experience requirement
DeTurris
Amadei4 at the University of Colorado Boulder describes a program in engineering for
developing communities to educate globally responsible students. Their vision calls on engineers
to not just solve problems of the one billion rich people on the planet, but also for the five billion
poor people. Johnson5 describes an engineering program as a collaboration between engineering
and language departments that includes an extended study and work abroad component. The
program‟s success is inspiring given that the university is small and has limited resources for
implementing new programs. Shuman6 summarizes the contributions of many different schools
in incorporating this component into their engineering program as it relates to the assessment of
professional skills in particular, which are more difficult.
Morell7 describes the impact on engineering education of a knowledge based economy from an
industry perspective. The key foundation for this work is to develop national/regional economic
development strategies worldwide. Recommendations on education reform include students
being active participants in their education process, faculty actively improving education, active
learning approaches including laboratories and internships, and an engineering curriculum that is
broad and flexible.
Global Competency Outcomes
Only one ABET Student Outcome of Criterion 3 specifically mentions the word „global‟.
Criterion 3h is defined as „the broad education necessary to understand the impact of engineering
solutions in a global, economic, environmental and societal context‟. The economic,
environmental and societal contexts are all components of a contemporary global perspective.
Three other outcomes lend themselves easily to an interpretation involving global issues. The
first is c, defined as „an ability to design a system, component or process to meet desired needs
within realistic constraints such as economic, environmental, social, political, ethical, health and
safety, manufacturability and sustainability‟. Realistic constraints include constraints imposed by
other cultures, natural resources and governments.
Another outcome is j, defined as „knowledge of contemporary issues‟ and the last is k, which is
„an ability to use techniques, skills and modern engineering tools necessary for engineering
practice‟. Contemporary issues in engineering encompass the need for solutions on a planet wide
scale, as detailed in the 14 Engineering Grand Challenges. Modern engineering tools necessary
for engineering practice include developments in capability by other countries that can‟t be
ignored in any industry that sells products to an international market.
Shuman describes assessment in terms of skills that are highly technical in nature, calling those
hard skills. Student Outcomes as 3a, b, c, e and k are listed as hard skills with the remainder of
the a-k outcomes falling under the heading of professional skills. These skills include 3d, f, g, h, i
DeTurris
and j. Shuman says that these professional skills are not assessed the same as the hard skills with
3h, i and j being defined as “awareness” outcomes. Those outcomes hope to capture the students‟
ability to know how to be aware of the importance of each and incorporate them into their
problem-solving activities6.
These three outcomes (h, i, j) focus on influencing engineering students‟ aims, attitudes and
values as they practice their engineering “hard” skills. These awareness skills center on the
students attitudes and values, therefore it makes sense to use behavioral observation as an
assessment technique for measuring frequency, duration, topology of student actions, usually in a
natural setting with non-invasive methods6.
At the start of EC2000, Prus and Johnson8 proposed six types of assessment methods for all of
(a-k) Student Outcomes. Although multiple methods should be used for measuring each
outcome, not all methods will work for the “awareness” oriented outcomes. The six assessment
methods are:
1)
2)
3)
4)
5)
6)
Tests and examinations
Measures of attitudes and perceptions
Portfolios
Performance appraisals and simulation
Behavioral observation
External examiner
Student Outcome h explicitly incorporates and c, j and k implicitly incorporate elements that can
be globalized. Two of these outcomes are hard skills (c and k), two are professional skills (h and
j). You can assess the global component of hard skills c and k. You can do that through an
assignment that has a global perspective. It becomes an adjustment to the scope of the program.
The professional skills are somewhat more energy intensive to assess. You have professional
skills, best understood as “attitudes”, and you assess attitudes using behavioral observation.
Three outcomes (h, i, j) focus on influencing engineering students‟ aims, attitudes and values as
they practice their engineering “hard” skills. How do you assess attitudes and values? Attitudes
are distinct from, but related to, behavior. You can see that in the results of an implicit
association test. Your attitudes influence but don‟t guarantee a certain behavior. It has been
shown that work sampling in some cases as a substitute for 100% behavior observation9.
Sampling theories have been extended to the observation of intervals that can capture,
accurately, the cognitive, behavioral and affective domains for processes that engineers normally
engage in, including working in teams, conducting design work and addressing ethical issues.
Levels of awareness in terms of testable skills allow for an assessment of the overall global
awareness, perspective and participation.
DeTurris
Global Competency Rubric
A rubric has been created that makes sense for accessing skills related to global competence and
can be categorized in terms of awareness, perspectives and participation. Lohmann doesn‟t talk
about assessment in terms of Student Outcomes. But they coursework, language and travel
components can be interpreted similarly to awareness, perspectives and participation.
Deardorff10 suggests assessment techniques for intercultural competence, although not
specifically for engineers. The pyramid model of Intercultural Competence shown in Figure 1
provides an excellent example of the progression of knowledge that is required for assessing
student competency in an intercultural environment.
Figure 1- Pyramid Model of Intercultural Competence10
The pyramid model includes a base of appropriate attitudes or awareness, with skills and
knowledge stemming from that. The desired internal outcome is an informed frame of reference
that creates a suitable perspective on intercultural issues. Finally, the desirable external outcome
becomes an active participation that enables successful intercultural interaction. These steps can
be employed to identify and access the progression of student learning for engineers.
The AAC&U adopts Bennett‟s11 Intercultural Knowledge and Competence Value Rubric that
connects knowledge, skills and attitudes first to a benchmark of awareness, then to several
DeTurris
milestones of perspectives and finally to a capstone of demonstration that includes articulating
insights and actively initiating participation with culturally different others. Intercultural
Knowledge and Competence is defined as “a set of cognitive, affective and behavioral skills and
characteristics that support effective and appropriate interaction in a variety of cultural contexts”.
Braskamp, et al12 defines student global learning and development in a similar way to
Deardorff‟s intercultural competence. Braskamp uses a scale spanning both the cognitive and
intrapersonal domains. Cognitive includes knowing and knowledge. The intrapersonal includes
identity, affect interaction and responsibility.
Downey et al describes a typology of methods for achieving global competency for engineers
that includes international enrollment, project, work placement and field trips, and integrated
class experience13. The learning criterion and outcomes are defined in Figure 2. These learning
outcomes suggest a scale of assessment that progresses from demonstrating knowledge or
awareness, to demonstrating ability to analyze having a more global perspective to finally to
displaying a disposition to value contributions of different perspectives and to bring those
perspectives into problem solutions.
By combining fundamental aspects of these definitions, a new compilation can be created. The
three simple themes that emerge for assessing international engagement include global
awareness, meaning recognition of the need to consider ourselves global citizens. The next level
of outcome is global perspectives, the ability to describe influence relationships between culture,
social, religious and linguistic differences between societies. The last level of outcome is global
participation, the direct connection of two cultures with intent to study differences in order to
solve larger problems within a larger framework. Global competence can be achieved at any of
these levels and can be assessed by implementing a standard rubric scale from basic knowledge
to exemplary achievement.
Learning Criterion
Through course instruction and interactions, students will acquire the
knowledge, ability and predisposition to work effectively with
people who define problems differently than they do.
Learning Outcomes
1.
Students will demonstrate substantial knowledge of the
similarities and differences among engineers and non-engineers
from different countries.
2.
Students will demonstrate an ability to analyze how people‟s
lives and experiences in other countries may shape or affect what
they consider to be at stake in engineering work.
3. Students will display a predisposition to treat co-workers from
other countries as people who have both knowledge and value,
may be likely to hold different perspectives than they do, and
may be likely to bring these different perspectives to bear in
processes of problem definition and problem solution.
Figure 2 – Minimum Learning Criterion and Learning Outcomes for Global Competency13
DeTurris
A rubric is presented in Table 1 that incorporates these levels of engagement for the ABET
criteria c, h, j and k. Each level, starting from a basic awareness to analyzing with perspective
and then incorporating the ideas into solutions increases the complexity of the cognitive
behaviors expected. The awareness aspect affects the underlying attitude that students will have
on this issue. It encompasses the ability to identify global factors. The perspective is a personal
understanding of how global issues will affect everyone, and having the skills and knowledge to
do something about it. Perspective includes an analysis of global factors. Participation is the
enactment of the attitude, skill and knowledge in a demonstrable form. The ability to apply the
analysis corresponds to a form of participation14. The understanding of each outcome shifts from
knowledge to analysis and then to a synthesis of the information into action.
Table 1: Rubric for ABET Student Outcomes c, h, j, and k
Awareness
Perspectives
Participation
c) Ability to design a system,
component or process to meet
desired needs within realistic
constraints such as economic,
environmental, social,
political, ethical, health and
safety, manufacturability and
sustainability.
Identifies realistic
constraints in a
global context
Determines realistic
constraints in a
global context
Assesses realistic
constraints in a
global context
h) Broad education necessary
to understand the impact of
engineering solutions in a
global, economic,
environmental and societal
context
Discusses impact of
engineering
solutions on global
context
Examines impact of
global perspective
on engineering
solutions
Justifies impact of
engineering
solution in a global
context
j) Knowledge of
contemporary issues
Describes
contemporary
issues in modern
global context
Distinguishes
contemporary
issues in modern
global context
Evaluates
contemporary
issues in modern
global context
k) Ability to use techniques,
skills and modern engineering
tools necessary for
engineering practice
Describes tools
necessary for
engineering practice
in a global context
Identifies tools
necessary for
engineering practice
in a global context
Incorporates tools
necessary for
engineering practice
in a global context
DeTurris
Conclusions
Globalization is now impacting engineering education in many ways. The education of students
includes a component of global competency, and the competency needs to be seen as having a
large number of assessment techniques. The ABET student outcomes Criterion 3, only mentions
the word global one time, but it can be easily interpreted as a component of three other
outcomes.
The student outcomes are broken down into two categories. The first category is hard, or
technical skills, the other category is professional skills. The student outcomes readily related to
global competence are c, j, h and k which include two technical skills and two professional skills.
The hard skills are assessed differently than professional skills, and professional skills like h in
particular require an assessment of “awareness” of an issue. The hard skills are assessed in the
usual methods, with the assignment including a global aspect to it that is imbedded into the
assignment. The professional skills need to be assessed with testing of attitudes and behavior
observation.
A rubric has been created for the engineering solutions in a global context of the student
outcomes c, h, j and k. The rubric describes competence in terms of awareness, perspective and
participation for each of the four relevant student outcomes. Assessment of global competence
for engineers can be made using all of the measurement methods proposed by Prus and Johnson
since two of the outcomes are hard skills and two are professional skills. The rubric uses a
progressive scale of examining, evaluating and incorporating skills as indicators of the level of
incorporation of each of the four criteria into global competency. The assessment of global
competency in engineering is not limited to measurements of student outcomes c, h, j and k. A
global component can be inferred in all of the other a-k student outcomes as well, and
widespread expansion of the definition of the expectations would go a long way towards
ensuring that all engineering programs educate their engineers for global competence.
References
1.
Falk, Dennis R., Moss, Susan, and Shapiro, Martin, “Educating Globally Competent Citizens”, published
by CSIS, April 2010.
2.
Grandin, John M., and Hirleman, E. Dan, “Educating Engineers as Global Citizens: A Call for Action/A
Report of the National Summit Meeting on the Globalization of Engineering Education”, Online Journal
for Global Engineering Education, Vol. 4, Issue 1, Article 1, 2009.
3.
Lohmann, Jack.R., Rollins, Howard A., Jr, and Hoey, J. Joseph, “Defining, developing and assessing global
competence in engineers”, European Journal of Engineering Education, ISSN 0304-3797 print, SEFI 2006.
DeTurris
4.
Amadei, B., “Program in Engineering for Developing Communities, Viewing the Developing World as the
Classroom of the 21st Century”, 33rd ASEE/IEEE Frontiers in Education Conference, Session F3B,
Boulder, CO, 2003.
5.
Johnson, E.W., and DeMaris, S.G., “Developing an International Engineering Experience for
Undergraduate Students at a Small Institution”, Online Journal for Global Engineering Education, vol 2,
issue 1, article 2, June 2007.
6.
Shuman, L.J., Besterfield-Sacre, M. and McGourty, J., “The ABET „Professional Skills‟ - Can They Be
Taught? Can They Be Assessed?” Journal of Engineering Education, 2005, 94(1), 41-55.
7.
Morell, Lueny, “Globalization and Engineering/Science Education: Do They Converge?” SEFI-IGIP Joint
Annual Conference, Miskolc, Hungary, July 2007.
8.
Prus, J., Johnson, R., “A Critical Review of student Assessment Option”, in Assessment and Testing:
Myths and Realities, Bers, T.H., and Mittler, M.L., eds, New Directions for Community Colleges, San
Francisco: Jossey-Bass, No. 88, Winter 1994, pp. 69-83.
9.
Yildirim, T.P., Townsend, J., Besterfield-Sacre, M., Shuman, L., Wolfe, H., “Developing Cognitive
Affective Behavioral Work Sampling Methodologies to Assess Student Learning Outcomes”, American
Society for Engineering Education, AC 2007-2435.
10. Deardorff, D.K., “Identification and Assessment of Intercultural Competence as a Student Outcome of
Internationalization”, Journal of Studies in International Education, 2006, 10:241.
11. Bennett, J.M., “Transformative training: Designing programs for culture learning”, in Moodian, M. (ed)
Contemporary Leadership and Intercultural Competence: Exploring the Cross-Cultural Dynamics Within
Organizations, Thousand Oaks, CA: Sage, 95-110, 2009.
12. Braskamp, L., Braskamp, D., & Merrill, K., “Assessing progress in global learning and development of
students with education abroad experiences”, Frontiers: The Interdisciplinary Journal of Study Abroad, ,
2009, vol 18, 101-118 ( http://www.frontiersjournal.com).
13. Downey, G.L., Lucena, J.C., Moskal, B.M., Parkhurst, R., Bigley, T., Hays, C., Jesiek, B.K., Kelly, L.,
Miller, J., Ruff, S., Lehr, J.L., Nicols-Belo, A., “The Globally Competent Engineer: Working Effectively
with People Who Define Problems Differently”, Journal of Engineering Education, April 2006.
14. http://www.cbe.csueastbay.edu/assessment/rubric/global.pdf