Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Department of History and Government Political Science B.A. in Political Science Program Code: 162 CIP Code: 45.1001 Program Quality Improvement Report 2009-2010 1 Student-Learning Outcomes Program Learning Outcomes Level of Thinking Skill Students completing the baccalaureate program in political science will demonstrate evaluation of theoretical and practical matters in the field of political science. Evaluation Students completing the baccalaureate program in political science will exhibit an ability to evaluate U.S. government and politics. Evaluation Students completing the baccalaureate program in political science will exhibit an ability to evaluate global politics. Evaluation Students completing the baccalaureate program in political science will exhibit an ability to evaluate scientific research and apply the scientific method. Evaluation Students completing the baccalaureate program in political science will demonstrate analytical and critical thinking skills. Evaluation Students completing the baccalaureate program in political science will demonstrate effective writing skills. Application Students completing the baccalaureate program in political science will develop basic computer skills. Application Program Quality Improvement Report 2009-2010 2 Alignment of Student-Learning Outcomes Cameron University Mission Statement • Political Science provides students with access to quality educational opportunities in and out of the classroom, to include: internships, guest speakers, independent and original research requirements, opportunities to engage in professional conferences, and educational travel. We produce students who think critically and are capable of answering both timeless questions and the challenges of the 21st century. • The Political Science faculty create a student-centered environment that combines innovative classroom teaching with experiential learning. We utilize simulations and interview opportunities, require original research, excellent writing and critical thinking, and offer opportunities for internship and travel to deepen the student experience. Program Quality Improvement Report 2009-2010 3 Alignment of Student-Learning Outcomes Cameron University Mission Statement • We prepare students for success by emphasizing critical thinking, diligent research, and strong writing. We encourage responsible citizenship not only among our own majors, but as part of our general education mission. Life-long learning is a necessary outcome of critical thinking, and while responsible citizenship is a meaningful contribution to our world, we also emphasize independent scholarship and thought, strong communication skills, to include computer, oral and writing skills, so that our students have the tools necessary to make a difference. • The Political Science Program is a driving force in the cultural life and economic development of the region through sponsorship of Constitution Day, public policy forums, election watches and commentary, world travel and through our students’ knowledge base. We seek to expand students’ horizons, and provide the tools necessary to achieve those far shores. Program Quality Improvement Report 2009-2010 4 Alignment of Student-Learning Outcomes Strategic Plan 2013 • Goal One of the Strategic Plan is that Cameron University be the University of Choice. This is consistent with the Political Science Goals which prioritize experiential learning, evaluation, critical thinking and writing. Examples of this include the degree of travel, exposure to national figures and the internships in which our students engage. • Goal Two of the Strategic Plan is that Cameron University provides the College Experience of Choice. We require that students work with faculty at the upper division levels as well as experience the practical elements of the field, through travel, conferences and electoral events. These requirements foster a sense of community among the students, faculty and alumni and encourage attention to cultural events in the wider community. • Goal Three of the Strategic Plan is that Cameron University be the Location of Choice. Through an emphasis on research, writing and critical thinking, as well as the sponsorship of public policy forums, conferences, and community electoral events, we believe that our faculty and students will effectively expand the number of educational, cultural, and social opportunities for the region. Program Quality Improvement Report 2009-2010 5 Alignment of Student-Learning Outcomes Strategic Plan 2013 • Goal Four of the Strategic Plan is that Cameron University is the Partner of Choice. The Political Science Program provides support for this goal through internships, community events, charitable activities and our educational emphases. Program Quality Improvement Report 2009-2010 6 Alignment of Student-Learning Outcomes School of Liberal Arts Mission • We offer a quality baccalaureate program in social science, which addresses all areas of the field of Political Science. We work to further the general education mission of the School through American Federal Government classes. We focus on a student-centered academic environment as we guide students to the highest possible standard of achievement, requiring reading, writing and critical thinking, as a liberal education is the mark of a free person. • Faculty scholarship and classroom application of scholarship is a priority. In addition, student clubs and numerous lectures, conferences and public forums, have allowed us to develop partnerships with the community, which include: academic conferences (OPSA), election forums, commentary and debates (KCCU and local political parties), Constitution Day (DAR), and policy forums (local business and civic leaders). Program Quality Improvement Report 2009-2010 7 Alignment of Student-Learning Outcomes Departmental Mission Statement • We support all the degree programs in Political Science, contribute to the degree in Social Studies Education, and offer a broad array of required and elective general education courses. • We engage energetically with students, other faculty, and the public as inspired teachers and dedicated researchers committed to fostering scholarship, dialogue, and service across the Cameron University community, Lawton/Fort Sill, and Southwest Oklahoma through our classrooms, our extensive club sponsorship, the sponsoring of public forums and travel. • We are passionate about political science, but our work crosses over into the areas of philosophy, public administration, and sociology and we are able to bring critical perspectives from those fields to all facets of our academic life. We have formed partnerships with elected officials, other universities, and outside faculty to achieve these goals. Program Quality Improvement Report 2009-2010 8 Measures of Learning and Service Outcomes Direct Measures Fall 2009 Program Learning Outcomes Level of Thinking Skill Direct Measures Students completing the baccalaureate program in political science will demonstrate evaluation of theoretical and practical matters in the field of political science. Evaluation MFT and Portfolio Students completing the baccalaureate program in political science will exhibit an ability to evaluate U.S. government and politics. Evaluation MFT and Portfolio Students completing the baccalaureate program in political science will exhibit an ability to evaluate global politics. Evaluation MFT and Portfolio Students completing the baccalaureate program in political science will exhibit an ability to evaluate scientific research and apply the scientific method. Evaluation MFT and Portfolio Students completing the baccalaureate program in political science will demonstrate analytical and critical thinking skills. Evaluation MFT and Portfolio Students completing the baccalaureate program in political science will demonstrate effective writing skills. Application Portfolio Students completing the baccalaureate program in political science will develop basic computer skills. Application Portfolio Program Quality Improvement Report 2009-2010 9 Measures of Learning and Service Outcomes Indirect Measures Fall 2009 Program Learning Outcomes Level of Thinking Skill Indirect Measures Students completing the baccalaureate program in political science will demonstrate evaluation of theoretical and practical matters in the field of political science. Evaluation Student Survey Students completing the baccalaureate program in political science will exhibit an ability to evaluate U.S. government and politics. Evaluation Student Survey Students completing the baccalaureate program in political science will exhibit an ability to evaluate global politics. Evaluation Student Survey Students completing the baccalaureate program in political science will exhibit an ability to evaluate scientific research and apply the scientific method. Evaluation Student Survey Students completing the baccalaureate program in political science will demonstrate analytical and critical thinking skills. Evaluation Student Survey Students completing the baccalaureate program in political science will demonstrate effective writing skills. Application Student Survey Students completing the baccalaureate program in political science will develop basic computer skills. Application Student Survey Program Quality Improvement Report 2009-2010 10 Measures of Student-Learning Outcomes - Strategies to Address Shortfalls • Mid Review: Following the completion of six upper division classes in political science, the relevant portfolio papers collected at that time will be systematically assessed by the political science faculty. • Assignments: Throughout the semesters, students taking political science classes are required to take a series of class-specific exams and writing assignments that require the application of the class material. Through written and oral feedback regarding students’ performance on these assignments, the faculty minimizes potential shortfalls in student learning prior to graduation on a systematic basis. Program Quality Improvement Report 2009-2010 11 Measures of Student-Learning Outcomes - Strategies to Address Shortfalls • Student Survey: During an academic year, usually during the spring semester, a survey to all political science majors is administered by the Political Science faculty. In addition to inquiring about students’ knowledge regarding U.S. Government, Comparative Politics, International Relations, and Research Methods, the survey asks a range of other program-related closed and openended question that allows the faculty to detect potential student learning shortfalls prior to graduation. Program Quality Improvement Report 2009-2010 12 Priority Learning Outcomes • The Political Science program’s assessment plan is built around three priority learning outcomes which are consistent with Cameron University’s Mission and Strategic plan and an analysis of the consumer need and interest. The prioritized program outcomes are: • • • Students completing the baccalaureate program in political science will demonstrate evaluation in the field of political science. Students completing the baccalaureate program in political science will demonstrate analytical and critical thinking skills. Students completing the baccalaureate program in political science will demonstrate effective writing skills. Program Quality Improvement Report 2009-2010 13 Report on actions from the three previously chosen priority outcomes • • The following priority-related measures, as agreed upon by the Political Science faculty during the previous PQIR cycle, was implemented: Concerning all three priority learning outcomes, the quantification of student scores, via a rubric, on portfolio papers was used. – 3 = exceeds expectations – 2 = meets expectations – 1 = does not meet expectations – Each paper type in portfolio has specific set of criteria. – The rubric was used by all political science faculty in evaluating the paper portfolios and the results are presented later in the presentation. Program Quality Improvement Report 2009-2010 14 Report on actions from the three previously chosen priority outcomes • Concerning the third learning priority outcome, “References” was added as an assessment criterion of the literature review. – References - citing the scholarly research literature using the APSA citation style. – The addition of references was added to the scoring rubric for the literature review paper and was scored by all political science faculty. Program Quality Improvement Report 2009-2010 15 Report on actions from the three previously chosen priority outcomes • Concerning all three priority learning outcomes, new criteria were established in the assessment of the critical theory paper in the portfolio. – Identification of the major arguments of the reading: Does the paper successfully identify the major arguments of the text on the topic assigned, both their conclusions and their reasons and supporting arguments? – Interpretive sensitivity and sophistication: Does the paper show familiarity with the details of the text and care and nuance in interpreting it? – Understanding of the significance of the author in question concerning the particular questions addressed in the assignment: Does the paper show understanding of what is unique and important in the author’s approach to the question being discussed? – The new criteria for the critical theory paper were used in assessing the student portfolios. Program Quality Improvement Report 2009-2010 16 Report on Actions - Student Learning Outcomes 2 and 3 1. The newly implemented quantitative assessment index of the portfolio papers for AY 2009-10, yield the following mean scores: Table 1: Mean Scores for Paper Portfolio Paper Mean Score Research Paper 2.7 Literature Review 1.8 Critical Theory Paper 2.8 Journal Article 2.5 Argumentative Brief 2.7 Program Quality Improvement Report 2009-2010 Where: 3 = Exceeds expectations 2 = Meets expectations 1 = Does not meet expectations 17 Learning Outcome 1 PROGRAM GOAL: Students completing the baccalaureate program in the field of political science will demonstrate higher order thinking skills MEASUREMENT OF LEARNING OUTCOME LEARNING OUTCOME 1 Students completing the baccalaureate program in political science will demonstrate evaluation of practical and theoretical matters in the field of political science. CURRICULUM AREA OR TARGET AUDIENCE All political science courses. Measurements MFT (direct), Portfolio (direct), Student Survey (indirect), Experiential Applications (indirect) Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements Content and face validity Established measures and intercoder reliability Program Quality Improvement Report 2009-2010 Schedule for measurements Spring Semester 18 Display of Assessment Data Major Field Test (MFT) • Political Science majors take the MFT in Political Science during the senior year. To demonstrate satisfactory performance on the MFT regarding evaluation in the field of political science, the program objective stipulates that Cameron University institutional mean score should be within one standard deviation of the national institutional mean score on the MFT assessment indicators. Six students took the MFT in the 2010 spring semester. Portfolio • Political Science majors turn in a series of papers as they proceed through the program. To demonstrate satisfactory performance regarding evaluation in the field of Political Science, the Political Science faculty has agreed on a set of paper-specific criteria that are widely used in the discipline. Both qualitative and quantitative approaches are used to evaluate the relevant student papers. Program Quality Improvement Report 2009-2010 19 Display of Assessment Data Student Survey • During the Spring semester of 2010, Political Science majors, consisting of sophomores, juniors, and seniors, evaluated their own perception of knowledge regarding some of the major sub-fields in Political Science (i.e. U.S. Government and Politics, Comparative Politics, and International Relations). Program Quality Improvement Report 2009-2010 20 Display of Assessment Data Measurement Participants MFT 6 Paper Portfolio Papers 23 Literature Review Research Papers Journal Article Critical Theory Argumentative Brief Student Survey Note: 4 4 7 3 5 16 Literature Reviews (Concepts of Political Science, PS 2113) Research Papers (Research Methods in Political Science, PS 2793) Journal Article Critiques (upper-division international relation/ comparative politics course) Critical Theory (upper-division political theory course) Argumentative Briefs (upper-division American government course) Program Quality Improvement Report 2009-2010 21 Analysis of Assessment Data Figure 1: 2006-10 Major Field Test Mean Score Comparison - Overall Knowledge 162 160 158 156 154 152 150 148 146 144 142 Cameron University Mean National Institutional Mean 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 22 Analysis of Assessment Data Figure 2: 2006-10 Major Field Test Mean Score Comparison - U.S. Government and Politics 60 Cameron University Mean National Institutional Mean 55 50 45 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 23 Analysis of Assessment Data Figure 3: 2006-10 Major Field Test Mean Score Comparison - Comparative Politics Cameron University Mean 60 National Institutional Mean 55 50 45 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 24 Analysis of Assessment Data Figure 4: 2006-10 Major Field Test Mean Score Comparison - International Relations 60 Cameron University Mean 55 National Institutional Mean 50 45 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 25 Analysis of Assessment Data Figure 5: 2006-10 Major Field Test Mean Score Comparison - Political Thought 55 Cameron University Mean 50 45 National Institutional Mean 40 35 30 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 26 Analysis of Assessment Data Figure 6: 2006-10 Major Field Test Mean Score Comparison - Methodology 80 Cameron University Mean 75 70 65 National Institutional Mean 60 55 50 45 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 27 Trend Analysis of Assessment Data MFT Scores • For the past four years, the Cameron University mean scores regarding the overall knowledge of political science have consistently stayed well above the national institutional mean and always within the required one standard deviation. Moreover, the overall trend indicate improvement from 149.0 for AY 2006-07 to 157.0 for AY 2009-10. These results indicate that Political Science majors at Cameron University are well above the National Institutional mean scores at 150.0 for AY 2009-10. • Except for Political Thought, a challenging and complex subject matter for students across the country, the remaining Political Science sub-field categories, including U.S. Government and Politics, Comparative Politics, International Relations, and Methodology, reiterate the previous patterns and clearly illustrate that Cameron University Political Science majors remain superior to their counterparts at the national institutional level. Program Quality Improvement Report 2009-2010 28 Trend Analysis of Assessment Data Paper Portfolio • Literature Review: Incorporated into the PQIR paper portfolio during the AY of 2008-09, the literature review shows that students are generally able to pose a research question, evaluate the literature and the relevant theoretical context, and point to further research. Some improvement could be sought in synthesizing the selected scholarly books and articles in terms of identifying common and contrasting themes or schools of thought. • Research Paper: Over the past five years, the research papers have continued to consistently improve in terms of students’ capacity to satisfactorily review the literature, discuss the research question and hypothesis, establish a coherent research design, and present the findings in a coherent fashion. Some improvements could be sought in the area of discussing the relevant theoretical context as it relates to the selected research topic. Program Quality Improvement Report 2009-2010 29 Trend Analysis of Assessment Data Paper Portfolio (continued) • Journal Article Critique: During the previous five years, the journal article critiques have consistently improved in the students’ ability to satisfactorily analyze the author’s thesis, the main arguments used by the author to support them, and the strengths and weaknesses of the article. While the critical evaluation of the selected journal articles has improved compared to previous years, there is still some room for improvement. • Critical Theory Paper: Over the past two years, the papers have consistently improved, illustrating the students’ ability to identify the major arguments of the reading, demonstrate interpretive sophistication, and show what is unique and significant about the author in question. As with the journal article reviews, critical evaluation of the selected theory articles has improved and it has now reached a level that is consistently acceptable across all the papers. Program Quality Improvement Report 2009-2010 30 Trend Analysis of Assessment Data Table 2: 2007-10 Student Survey of Political Science Majors N = 12 (2007-08) N = 26 (2008-09) N = 16 (2009-10) My knowledge concerning U.S. government and politics has increased. My knowledge concerning global issues has increased. My knowledge concerning the research process in political science has increased Strongly Agree Agree Neither Agree/ Disagree Disagree Strongly Disagree 67.00% (8) 33.00% (4) 0.00% (0) 0.00% (0) 0.00% (0) 76.90% (20) 23.10% (6) 0.00% (0) 0.00% (0) 0.00% (0) 2008-09 62.50% (10) 31.25% (5) 6.25% (1) 0.00% (0) 0.00% (0) 2009-10 67.00% (8) 16.50% (2) 16.50% (2) 0.00% (0) 0.00% (0) 57.70% (15) 30.70% (8) 11.60% (3) 0.00% (0) 0.00% (0) 56.25% (9) 31.25% (5) 12.50% (2) 0.00% (0) 0.00% (0) 42.00% (5) 42.00% (5) 16.00% (2) 0.00% (0) 0.00% (0) 38.40% (10) 46.20% (12) 7.70% (2) 7.70% (2) 0.00% (0) 62.5% (10) 37.5% (6) 0.00% (0) 0.00% (0) 0.00% (0) Program Quality Improvement Report 2009-2010 2007-08 31 Trend Analysis of Assessment Data Figure 7: 2007-10 Student Survey of Political Science Majors - Knowledge of U.S. Government and Politics Increased 90 80 Strongly Agree 70 Agree 60 50 40 30 20 10 0 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 32 Trend Analysis of Assessment Data Figure 8: 2007-10 Student Survey of Political Science Majors - Knowledge of Global Issues Increased - Program Quality Improvement Report 2009-2010 33 Trend Analysis of Assessment Data Figure 9: 2007-10 Student Survey of Political Science Majors - Knowledge of Research Process Increased - Program Quality Improvement Report 2009-2010 34 Trend Analysis of Assessment Data • Student Survey • • • • Utilizing the Student Surveys for Political Science for AYs 2007-08 and 2009-10, fifty-four majors, consisting of sophomores, juniors, and seniors, evaluated the program very favorably. Reiterating the trends established by the respective MFT categories, the student survey results indicate: Knowledge concerning U.S. politics/government • Remained steady across the Strongly Agree and Agree categories combined at 100 percent. Knowledge concerning global issues • Increased across the Strongly Agree and Agree categories combined from 83.5 in AY 2007-08 percent to 87.5 percent in AY 2009-10 Knowledge concerning research process • Increased across the Strongly Agree and Agree categories combined from 84.0 percent in AY 2007-09 to 100.0 percent in AY 2009-10 Program Quality Improvement Report 2009-2010 35 Action Plan 1. Although we do not have sufficient faculty to offer our courses on a shorter rotation (as several respondents to the Student Survey requested especially in regards to the Research Methods in Political Science - PS 2793), we plan to further strengthen students’ evaluation in the field of Political Science and especially with respect to global issues. Accordingly, we continue to plan the establishment of a Model UN club over the next 12-18 months. No additional human/fiscal resources are required to implement this action. 2. Recognizing the relatively minor weaknesses in regards to the paper portfolio, the Political Science faculty will continue its efforts to refine the specific paper criteria and clearly convey those to the students. No additional human/fiscal resources are required to implement this action. Program Quality Improvement Report 2009-2010 36 Learning Outcome 2 PROGRAM GOAL: Students completing the baccalaureate program in political science will demonstrate a high level of analytical and critical thinking skills. MEASUREMENT OF LEARNING OUTCOME LEARNING OUTCOME 2 Students completing the baccalaureate program in political science will demonstrate analytical and critical thinking skills. CURRICULUM AREA OR TARGET AUDIENCE Upper-division political science courses Measurements Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements MFT (direct) and Portfolio (direct) Content and face validity Established measures and intercoder reliability Program Quality Improvement Report 2009-2010 Schedule for measurements Spring Semester 37 Display of Assessment Data Major Field Test (MFT) • Political Science majors take the MFT in Political Science during the senior year. To demonstrate satisfactory performance on the MFT regarding analytical and critical thinking, the program objective stipulates that Cameron University institutional mean score should be within one standard deviation of the national institutional mean score on the MFT assessment indicators. Six students took the MFT in the 2010 spring semester. Portfolio • Political Science majors turn in a series of papers as they proceed through the program. To evaluate satisfactory performance regarding analytical and critical thinking, the Political Science faculty focuses on measurements that are part of each portfolio paper and derives an overall qualitative and quantitative assessment of the papers accordingly. Program Quality Improvement Report 2009-2010 38 Learning Outcome 2 Measurement Participants MFT 6 Paper Portfolio Papers 23 Literature Review Research Papers Journal Article Critical Theory Argumentative Brief Note: 4 4 7 3 5 Literature Reviews (Concepts of Political Science, PS 2113) Research Papers (Research Methods in Political Science, PS 2793) Journal Article Critiques (upper-division international relation/ comparative politics course) Critical Theory (upper-division political theory course) Argumentative Briefs (upper-division American government course) Program Quality Improvement Report 2009-2010 39 Trend Analysis of Assessment Data Figure 2: 2006-10 Major Field Test Mean Score Comparison - Critical and Analytical Thinking 90 80 70 60 50 40 30 20 10 0 Cameron University Mean National Institutional Mean 2006-07 2007-08 2008-09 2009-10 Program Quality Improvement Report 2009-2010 40 Trend Analysis of Assessment Data MFT Scores • For AYs 2008-09 and 2009-10, the Cameron University mean scores regarding analytical and critical thinking were within one standard deviation of the National Institutional mean. Similar to the previous MFT measurements, the analytical and critical thinking skills of Political Science majors for AY 2009-10 at 78 are well above the National Institutional mean score of 69.2. Portfolio • Overall, the portfolio papers including the literature reviews, research papers, journal article reviews, critical theory papers, and argumentative briefs indicate that Cameron Political Science majors are able to critically analyze complex subject matters in the various sub-fields of Political Science. While performance levels have clearly increased, some improvement could be sought in the area of critically evaluating the major arguments, especially in relation to the journal article reviews and critical theory papers. Program Quality Improvement Report 2009-2010 41 Action Plan 1. Recognizing the relatively minor weaknesses in regards to the specific aspects of critical thinking as they relate to the paper portfolio, the Political Science faculty will continue its efforts to refine the specific paper criteria and clearly convey those to the students. No additional human/fiscal resources are required to implement this action. Program Quality Improvement Report 2009-2010 42 Learning Outcome 3 PROGRAM GOAL: Students completing the baccalaureate program in political science will demonstrate a high level of writing skills. MEASUREMENT OF LEARNING OUTCOME LEARNING OUTCOME 3 Students completing the baccalaureate program in political science will demonstrate effective writing skills. CURRICULUM AREA OR TARGET AUDIENCE Concepts of Political Science (PS 2113) Measurements Methods used to determine validity of measurement instruments Methods used to determine reliability of measurements Portfolio Face validity Intercoder reliability Schedule for measurements Spring Semester Research Methods (PS 2793) Upper-division political science courses (comparative politics, international relations, American government, and theory) Program Quality Improvement Report 2009-2010 43 Trend Analysis of Assessment Data Portfolio • Overall, the portfolio papers including the literature reviews, research papers, journal article reviews, critical theory papers, and argumentative briefs indicate that Cameron Political Science majors demonstrate effective writing – a critical skill that has also improved over the years. Program Quality Improvement Report 2009-2010 44 Trend Analysis of Assessment Data Measurement Participants Paper Portfolio Papers 23 Literature Review Research Papers Journal Article Critical Theory Argumentative Brief Note: 4 4 7 3 5 Literature Reviews (Concepts of Political Science, PS 2113) Research Papers (Research Methods in Political Science, PS 2793) Journal Article Critiques (upper-division international relation/ comparative politics course) Critical Theory (upper-division political theory course) Argumentative Briefs (upper-division American government course) Program Quality Improvement Report 2009-2010 45 Action Plan 1. The Political Science faculty’s will continue its successful efforts to teach effective writing skills. No additional human/fiscal resources are required to implement this action. Program Quality Improvement Report 2009-2010 46 Published information on graduates Academic Year 09-10 Entered Graduate School Working In Discipline Other 1 1 Summer 2009 Fall 2009 Spring 2010 2 3 Total 2 4 Program Quality Improvement Report 2009-2010 1 47