UNF-MPA student learning outcomes and program assessment

UNF-MPA student learning outcomes and program assessment Prepared by Dr. G.G. Candler MPA Director 7 May 2014 UNF MPA program assessment includes a number of elements: 1. An alumni survey, 2. a student skills self-assessment, 3. a civic literacy quiz, 4. a public management writing exercise, 5. a quantitative analysis exercise, 6. a concentration assessment exercise, and 7. student numbers and academic qualifications. We also add a sixth section, which presents program student numbers and admissions information. Page 1 of 12

1. Alumni survey A first survey of UNF-MPA alumni was conducted in Spring 2012. The survey used the Vovici Enterprise online survey platform. The survey url was mailed out by Alumni Affairs (they do not share addresses of program alumni with programs), to the 253 UNF-MPA alumni they have contact information for. 44 useful responses were received, half of which (22) work in Duval County, four more elsewhere in northeast Florida, and six Florida, with the county not specified. Employment sector varied, with nine in state government, eight in the nonprofit sector, seven each in federal, local government and the private sector, and four retired. As indicated in Table 1, respondents reported strong overall impressions of the program. Table 1 Overall alumni impression of MPA Yes Improvement in life satisfaction 84% Worth the investment in time and money 93% More specifically, the survey gathered information regarding increased income as a result of the MPA. This data is presented in Table 2, below. The data indicates strong increases in income for those who responded to the survey. Table 2 Increase in income after MPA graduation Increase in annual salary one year after five years after 1980-96 N 11 11 Mean ($) 9,709 26,182 Median ($) 5,300 25,000 1999-2004 N 11 11 Mean ($) 12,818 37,273 Median ($) 5,000 30,000 2006-11 N 10 5 Mean ($) 20,650 29,001 Median ($) 18,000 12,000 Total N 32 27 Mean ($) 14,197 31,222 Median ($) 7,650 25,000 Note: The standard error of the estimates are under $3000 for the mean salary increase after one year, and just over $5000 for the five year figure. Finally, the survey also gathered data on professional advancement. This is presented in Table 3. While over-analysis of these results should be avoided, given the relatively small sample size, the program has seen a number of graduates enter executive level positions. Page 2 of 12

2. Student skills self-assessment. Table 3 Alumni employment status Entering MPA Current employment status status Executive 3 8 Management 18 18 Professional 0 2 Self-employed 0 2 Technician/analyst 12 8 Other (line, student) 11 6 In the student skills self-assessment, skills surveyed for are those identified in the program s mission and competencies. This has been applied to incoming students, in the introductory, PAD6060 Public Administration in Modern Society class, graduating students, in the PAD6060 Capstone Seminar course, and alumni. The questions were coded as shown in Table 4. Table 4 Coding of the Student Self-Assessment Survey Intro Capstone 0 Unchanged 1 Need no improvement 1 Unchanged, prior experience left little room to improve 2 Need improvement 2 Improved 3 Need significant improvement 3 Improved significantly The entrance survey measures the perceived professional development needs of incoming students; while the exit survey measures the perception of Capstone students regarding their development through the MPA program. The dimensions on which the students are rated are derived from goals based on the program mission, though as this was just recently revised, some modifications have been made to the instrument. For the Capstone class, students are asked to rate their development in terms of whether their skill level was unchanged, improved, or improved significantly. For students who indicated that they remain unchanged on an assessment criterion, they were asked why, as a number of students indicate that their prior experience leaves them little room to improve. Results are presented in Tables 5, 6, and 7. Page 3 of 12

Table 5 Student skill self-assessment, pre-test (Introductory PAD6060) Student needs no improvement improvement significant improvement Communication: -- writing 15.4 65.4 19.2 -- speaking 19.2 60.6 20.2 -- listening 26.9 56.7 16.3 Local governance 18.4 43.7 37.9 Problem solving 12.6 67.0 20.4 Professional ethics 24.3 52.4 23.3 Management theory 9.7 69.9 20.4 Economic constraints 13.5 51.0 3 Quantitative skills 16.3 62.5 21.2 Diverse workforce 37.5 46.2 16.3 Public policy* 8.1 71.0 21.0 Globalization 17.3 56.7 26.0 Role of public service* 9.7 64.5 25.8 Concentration expertise 18.6 55.9 25.5 Notes: * Sample size 104, except for those with an asterisk, which had 62. Interpretation of these results is a bit challenging, though note especially that many students have had a high impression of their skill levels on entering the program, with between 8-38% of incoming students reporting that they need no improvement, and generally under a third of students reporting that they need significant improvement. As shown in Table 6 below, though, on exiting the program a large portion of students report improvement, or significant improvement in skill levels. The program has performed especially well in terms of understanding the role of public service, local governance, and their concentration area of expertise; and has also been strong in professional ethics and economics. Page 4 of 12

Table 6 Student skill self-assessment: post-test (Capstone PAD6066) Student skills were Communication: -- writing 2010-12 -- speaking 2010-12 -- listening 2010-12 Local governance 2010-12 Problem solving 2010-12 Professional ethics 2010-12 Management theory 2010-12 Economic constraints 2010-12 Quantitative skills 2010-12 Diverse workforce 2010-12 Public policy Spring 12 Globalization 2010-12 Role of public service Spring 12 Concentration expertise Class Unchanged Unchanged (prior exp.) 2010-12 5.4 3.4 5.4 3.4 8.9 1.8 11.1 5.4 3.6 3.4 3.6 8.9 6.9 3.6 7.1 3.4 3.6 7.1 16.1 20.7 10.7 25.0 1.8 5.4 7.1 11.1 1.8 7.1 14.3 1.8 3.4 Improved 57.1 44.4 55.2 57.1 61.1 55.2 58.9 61.1 5 46.4 38.9 34.5 55.4 5 69.0 5 33.3 41.4 50.9 44.4 69.0 53.6 44.4 42.9 55.4 5 48.3 60.7 61.1 53.6 5 44.4 51.7 48.2 5 72.4 33.3 5 34.5 46.4 38.9 31.0 Improved significantly 30.4 5 41.4 21.4 27.8 20.7 21.4 27.8 25.0 5 5 65.5 33.9 38.9 31.0 37.5 5 55.2 49.1 5 31.0 44.6 5 53.6 28.6 44.4 44.8 35.7 27.8 25.0 44.4 5 44.8 46.4 5 27.6 66.7 5 65.5 53.6 61.1 65.5 Page 5 of 12

As indicated, the survey was also carried out on alumni in 2012. Respondents were asked to assess their skill development as a result of the MPA program. Alumni self-assessment results are presented in Table 7. Table 7 Alumni perceptions of skill assessment Unchanged Alumni skills were Improved Unchanged (prior exp.) Improved significantly Communication: -- writing 11.4 9.1 27.3 52.3 -- speaking 9.1 6.8 70.5 13.6 -- listening 15.9 6.8 59.1 18.2 Local governance 6.8 43.2 47.7 Problem solving 2.3 4.5 45.5 47.7 Professional ethics 4.5 13.6 5 31.8 Management theory 6.8 2.3 54.5 36.4 Economic constraints 4.5 4.5 56.8 34.1 Quantitative skills 6.8 9.1 40.9 43.2 Diverse workforce 13.6 9.1 59.1 18.2 Public policy 4.5 52.3 43.2 Globalization 20.5 4.5 52.3 22.7 Role of public service 2.3 4.5 56.8 36.4 Concentration expertise 4.5 56.8 38.6 Change in life satisfaction 9.8 n/a 56.1 34.1 The results are strong across the board. Only in an understanding of globalization (which has not been part of the curriculum, or mission prior to 2011), diversity, and perhaps communication skills, have there been historical weaknesses. Yet at least 75%, and as many as 95% of respondents reported improvement on all items. 3. Student Civic Literacy quiz The Student Civics quiz is based on a national survey, which allows for comparison outside the program. The survey consists of a number of Constitutional, Historical and Economic questions. The survey seeks to test whether the program has provided a broad-based understanding of the context of governance. Results are presented in Table 5. The process used in this evaluation is also a pre-test (PAD 6060 Intro) and post-test (PAD 6066 Capstone). Students complete this quiz anonymously, and are not graded on it. As of Spring 2014, we have applied the quiz to six Intro classes (n = 96), and four Capstone classes (n = 107). Prior to this semester, results have been mixed. While UNF-MPA Capstone students have consistently outperformed incoming students, these differences have not been statistically significant. As we gathered more data it was expected that this would change, and we reached that point in Spring 2014. As can be seen, the number of correct answers (out of 33 possible) is Page 6 of 12

1.4 higher for Capstone students, at a 95% level of statistical confidence (p =.039). In the subcategories, the Constitutional questions are especially higher (by 6.1%, p =.015) for Capstone students, while the other two categories have lower levels of statistical significance. Table 8 Student civic literacy survey Constitutional History Economic Civic literacy (% correct) #/33 Intro (n = 96) 70.7 63.4 59.6 21.7 Capstone (n = 107) 76.8 66.5 64.3 23.1 (p =.015) (p =.169) (p =.107) (p =.039) US undergrads (n = 587) 18.8 Intro (n = 96) 21.7 (p =.000) US grads (n = 412) 21.1 Capstone (n = 107) 23.1 (p =.000) The UNF MPA program also consistently outperforms the broader population, both on entering and leaving the MPA program. Though a fairly quixotic assessment tool, the logic behind it has been to test to see if, as a result of time in the program, students develop a richer understanding of the broader context in which public managers operate, despite our not specifically teaching either these topics, or the specific questions in the quiz. 4. Public Management essay The third assessment exercise applied by the UNF-MPA program is a public management writing exercise, also applied as a pre-test (PAD 6060) and post-test (PAD 6066). To date the essay has been taken by three Introductory class, and the four Capstone classes reported. Students write an essay on similar articles in each exercise, and are assessed on the following Learning Outcomes: Discipline knowledge. Demonstrate command of the public management literature. Critical thinking. Demonstrate the ability to make a logical, coherent, balanced argument. Research. Demonstrate the ability to gather evidence. Writing. Demonstrate written communication competence. Each learning outcome is scored for each student on a 0-3 scale, as shown: 3 Mastery, above normal requirements for professional work. 2 At a level appropriate for professional work. 1 Insufficient for professional work, requires minor improvement. 0 Insufficient for professional work, requires considerable improvement. Table 9-11 present the results. Table 9 presents results by the percentage of students who fall in each rubric category for each learning outcome. With one exception (the research score for the Spring 2011 Capstone), each Capstone class outscored the combined scores for the Introductory classes. Page 7 of 12

Discipline knowledge Critical analysis Research Writing Table 9 Writing exercise, by learning outcomes Class Insufficient major Insufficient minor Prof. Adequate Prof. Mastery Mean Score Intro- 24.6 10.5 21.1 43.9 1.84 Caps-Spg 11 0 21.1 10.5 68.4 2.47 Caps-Spg 12 4.8 4.8 14.3 76.2 2.62 Caps-Spg 13 18.5 14.8 7.4 59.3 2.07 Caps-Spg 14 2.9 17.1 28.6 51.4 2.29 Intro- 42.9 4.8 14.3 38.1 1.84 Caps-Spg 11 0 0 31.6 68.4 2.68 Caps-Spg 12 4.8 4.8 19.0 71.4 2.57 Caps-Spg 13 25.9 11.1 7.4 5 1.93 Caps-Spg 14 2.9 11.4 2 65.7 2.49 Intro- 19.0 19.0 14.3 47.6 2.04 Caps-Spg 11 10.5 42.1 21.1 26.3 1.63 Caps-Spg 12 0 4.8 33.3 61.9 2.57 Caps-Spg 13 22.2 7.4 11.1 59.3 2.07 Caps-Spg 14 5.7 11.4 17.1 65.7 2.43 Intro- 19.0 14.3 28.6 38.1 2.09 Caps-Spg 11 15.8 5.3 31.6 47.4 2.11 Caps-Spg 12 9.5 4.8 33.3 52.4 2.29 Caps-Spg 13 11.1 11.1 7.4 70.4 2.37 Caps-Spg 14 2.9 5.7 25.7 65.7 2.54 Table 10 combines these scores for each individual student. So while Table 9 combines the scores of students for each learning outcome, Table 10 indicates the combined (the four Learning Outcomes averaged) performance of individual students. Each cell presents the number of students whose average score on the four Learning Outcomes was in each of these ranges. The four categories are broadly equivalent to the four categories used in Table 9: an overall score of 0-1 equates to major insufficiency, 1.25 to 1.75 minor insufficiency, 2 to 2.25 professional sufficiency, and 2.5+ professional mastery. Table 10 Writing exercise, by average student score on all four items. Class Class Frequency distribution, average score Class Mean Median 0 1.0 1.25 1.75 2.0 2.25 2.5 3.0 Intro- Caps-Spg 11 Caps-Spg 12 1.95 2.22 2.52 2.25 2.25 2.75 15 1 2 9 3 1 6 8 2 27 7 16 Caps-Spg 13 2.11 3.00 6 2 2 16 Caps-Spg 14 2.45 2.75 1 6 3 25 The Table demonstrates strong improvement, with about half of incoming students having professionally insufficient writing skills, and this percentage dropping to under 20% by the Capstone class. That about 20% of Capstone students are unable to write to a standard adequate Page 8 of 12

for a professional environment, though, suggests that this is an area that the program needs to work on. Finally, Table 11 presents tests of statistical significance between the Introductory (n = 57) and Capstone (n = 102) students. All scores are higher for the Capstone class. Except for the quality of Research in these short papers, the other four differences are all statistically significant: weak in the case of Writing, but very strong in Discipline Knowledge, Critical Analysis, and Combined. Table 11 Intro and Capstone compared, writing exercise Assessment Class Mean Probability item score Discipline Intro 1.84.006 knowledge Capstone 2.33 Critical analysis Intro 1.84.003 Capstone 2.39 Research Intro 2.04.297 Capstone 2.22 Writing Intro 2.09.091 Capstone 2.36 Combined Intro Capstone 1.95 2.33.011 5. Quantitative analysis exercise The UNF-MPA program adopted a fourth major assessment item in Spring : an assessment of student quantitative analysis skills. While Table 6 shows that nearly all Capstone students have subjectively indicated that their quantitative skills were improved, this exercise applies an objective test to that assessment. Students were presented with raw data on education and income in Florida and Indiana counties. Students were also presented with SPSS correlation and regression output using that data, asked to interpret this data, and present a table with education and income figures for a small number of northeast Florida counties. Students were assessed on their knowledge of quantitative analysis, their critical analytical skills, and their table presentation. Results are presented in Table 10. Table 12 Quantitative analysis exercise, by Capstone learning outcomes (%, and on 0-3 scale for mean score) Statistical interpretation (n = 25) 2014 (n = 33) Critical analysis 2014 Table presentation 2014 Combined 2014 Insufficient major 32.0 6.1 2 6.1 4.0 18.7 4.0 Insufficient minor 4 39.4 36.0 30.3 2 3.0 32.0 24.2 Prof. Adequate 16.0 24.2 32.0 24.2 4 3.0 29.3 17.1 Prof. Mastery 12.0 30.3 12.0 39.4 36.0 93.9 2 54.5 Mean Score 1.08 1.79 1.36 1.97 2.08 2.91 1.51 2.23 Page 9 of 12

6. Concentration assessments. The program also administers a Local Government competence essay, and a Concentration Competence essay. These exercises have been carried out (and reported) for Spring, and are in progress for Spring 2014. However while initial results are good, numbers are small, so take the Concentration assessment results in Table 13 as preliminary. Table 13 Concentration assessment exercise, by Spring -14 Capstone competency Insufficient major Insufficient minor Prof. Adequate Prof. Mastery Nonprofit concentration 1 Critical policy thinking (#) Mean score 0 2 1 11 2.64 0 14.3 7.1 78.6 Management literature (#) 1 1 1 11 2.57 7.1 7.1 7.1 78.6 Combined 3.5 10.7 7.1 78.6 2.61 Local government concentration Critical policy thinking (#) 1 0 2 2 2.00 2 0 4 4 Management literature (#) 0 1 2 2 2.20 0 2 4 4 Combined 0 16.7 5 33.3 2.10 Health admin concentration Critical policy thinking (#) 2 0 1 2 1.60 4 2 4 Management literature (#) 0 3 1 1 1.60 0 6 2 2 Combined 2 3 2 3 1.60 Public Policy concentration 2 Public policy literature (#) 0 1 0 4 2.60 0 2 0 8 Critical policy thinking (#) 0 0 1 4 2.80 (#) 0 0 2 8 Combined 0 1 1 8 2.70 Conflict transformation Applied ethics These two independent programs are concentration options for MPA students. To date no student has completed either. Notes: 1 Double graded. The others have not been to date. 2 -- Spring 2014 only, as this concentration was added in Fall 2012, and the first graduate occurred in Fall. Page 10 of 12

7. Program student numbers and academic qualifications Recent student numbers have returned to levels seen prior to the 2009-11 recession. The total number of enrolled students (see Table 14) in the program has varied from 90-100+ over the past couple of years. Enrollments (total class enrollments) have also returned to their pre-recessionary levels. This is shown in Table 15, below. For the past few years, we have offered 20 classes a year: eight each in fall and Spring, and four in the summer, though are offering seven in Spring 2014. We expect a decrease in student numbers in the short term, as a too large (37 student) Spring 2014 Capstone class graduates over the next year. Table 14 Enrolled student numbers 2011-12 2012-13 -14 MPA Fall 62 71 67 Spring 69 83 77 NP Cert Fall 17 18 20 Spring 18 16 24 Post-bac Fall 6 2 5 Spring 4 1 2 Total Fall 85 91 92 Spring 91 100 103 Table 15 Total class enrollments, and graduates 2006-7 2007-8 2008-9 2009-10 2010-11 2011-12 2012-13 -14 Class totals a Fall 151 172 147 214 193 157 176 164 Spring 149 133 155 198 187 170 200 170 b Summer 68 88 109 109 85 63 85 --- Total 368 393 411 511 465 390 461 --- Graduates Total 26 33 17 29 31 27 25 --- Notes: a b Total enrollments in all courses, as opposed to the Table 3 number of individual students in the program. Application periods are set by the Graduate College, and refer to the semester of admission Page 11 of 12

Table 16 presents recent data on enquiries, applications, and admission decisions. Numbers appear to be holding steady. Spring 2011 Sum 2011 Table 16 Enquiries, applications and admissions Fall Spring Sum Fall 2011 2012 2012 2012 Spring Sum Fall Enquiries a MPA 65 53 68 65 65 52 57 75 65 --- NP cert 23 19 16 28 24 44 22 17 8 --- MPA App ns b - 12 9 27 23 8 35 20 3 20 18 Admit 5 6 16 12 5 22 12 0 17 13 Provisional 3 0 5 6 2 6 3 1 2 3 Deny 3 1 5 3 0 6 2 2 1 1 Others c 1 2 1 2 1 1 3 0 0 1 Enroll d 6 25 16 1 12 16 NP cert -- App ns 2 5 13 9 7 13 2 8 10 8 Admit 2 5 11 6 7 12 2 6 10 8 Deny 0 0 2 3 0 1 0 2 0 0 Spring 2014 Notes: a Enquiries from September-December for Fall; January-April for Spring; May-August for Summer. The numbers represent individual prospects who contacted the program during the period in question. b Application periods are set by the Graduate College, and refer to the semester of admission c d Readmitted students, and successful post-baccalaureate students. This is data from the Graduate School. Table 17 presents applicant academic qualifications. Note that MPA admission requirements are a 3.0 GPA, and combined GRE score of 1000 (on the old measurement system). There has been a slight drop in these credentials, as a result of the admission of 2-3 students with low academic credentials, but strong work experience or successful completion of classes as Post- Baccalaureate students. Spg 2011 Table 17 Applicant academic qualifications Fall Spg Sum Fall 2011 2012 2012 2012 Sum 2011 Spg Sum Fall Spg 2014 MPA Admit GPA 3.50 3.27 3.25 3.44 3.17 3.31 3.48 --- 3.45 3.35 GRE 966 1058 1113 1063 1095 1061 1115 --- 1091 1053 Provisional GPA 3.08 --- 2.83 3.05 2.94 2.84 3.11 3.14 3.04 3.00 GRE 827 --- 828 778 930 968 860 875 870 845 Deny GPA 2.88 2.96 2.51 2.49 --- 2.75 2.65 2.36 2.64 3.40 GRE 823 690 760 793 --- 695 935 860 800 570 NP cert. Admit GPA n/a n/a 3.18 3.29 3.24 3.37 3.56 3.36 3.48 3.19 Deny GPA --- --- 2.43 2.67 --- 2.12 --- 2.56 --- --- Page 12 of 12