UNF-MPA student learning outcomes and program assessment

Transcription

1 UNF-MPA student learning outcomes and program assessment Prepared by Dr. G.G. Candler MPA Director 7 May 2014 UNF MPA program assessment includes a number of elements: 1. An alumni survey, 2. a student skills self-assessment, 3. a civic literacy quiz, 4. a public management writing exercise, 5. a quantitative analysis exercise, 6. a concentration assessment exercise, and 7. student numbers and academic qualifications. We also add a sixth section, which presents program student numbers and admissions information. Page 1 of 12

2 1. Alumni survey A first survey of UNF-MPA alumni was conducted in Spring The survey used the Vovici Enterprise online survey platform. The survey url was mailed out by Alumni Affairs (they do not share addresses of program alumni with programs), to the 253 UNF-MPA alumni they have contact information for. 44 useful responses were received, half of which (22) work in Duval County, four more elsewhere in northeast Florida, and six Florida, with the county not specified. Employment sector varied, with nine in state government, eight in the nonprofit sector, seven each in federal, local government and the private sector, and four retired. As indicated in Table 1, respondents reported strong overall impressions of the program. Table 1 Overall alumni impression of MPA Yes Improvement in life satisfaction 84% Worth the investment in time and money 93% More specifically, the survey gathered information regarding increased income as a result of the MPA. This data is presented in Table 2, below. The data indicates strong increases in income for those who responded to the survey. Table 2 Increase in income after MPA graduation Increase in annual salary one year after five years after N Mean ($) 9,709 26,182 Median ($) 5,300 25, N Mean ($) 12,818 37,273 Median ($) 5,000 30, N 10 5 Mean ($) 20,650 29,001 Median ($) 18,000 12,000 Total N Mean ($) 14,197 31,222 Median ($) 7,650 25,000 Note: The standard error of the estimates are under $3000 for the mean salary increase after one year, and just over $5000 for the five year figure. Finally, the survey also gathered data on professional advancement. This is presented in Table 3. While over-analysis of these results should be avoided, given the relatively small sample size, the program has seen a number of graduates enter executive level positions. Page 2 of 12

3 2. Student skills self-assessment. Table 3 Alumni employment status Entering MPA Current employment status status Executive 3 8 Management Professional 0 2 Self-employed 0 2 Technician/analyst 12 8 Other (line, student) 11 6 In the student skills self-assessment, skills surveyed for are those identified in the program s mission and competencies. This has been applied to incoming students, in the introductory, PAD6060 Public Administration in Modern Society class, graduating students, in the PAD6060 Capstone Seminar course, and alumni. The questions were coded as shown in Table 4. Table 4 Coding of the Student Self-Assessment Survey Intro Capstone 0 Unchanged 1 Need no improvement 1 Unchanged, prior experience left little room to improve 2 Need improvement 2 Improved 3 Need significant improvement 3 Improved significantly The entrance survey measures the perceived professional development needs of incoming students; while the exit survey measures the perception of Capstone students regarding their development through the MPA program. The dimensions on which the students are rated are derived from goals based on the program mission, though as this was just recently revised, some modifications have been made to the instrument. For the Capstone class, students are asked to rate their development in terms of whether their skill level was unchanged, improved, or improved significantly. For students who indicated that they remain unchanged on an assessment criterion, they were asked why, as a number of students indicate that their prior experience leaves them little room to improve. Results are presented in Tables 5, 6, and 7. Page 3 of 12

4 Table 5 Student skill self-assessment, pre-test (Introductory PAD6060) Student needs no improvement improvement significant improvement Communication: -- writing speaking listening Local governance Problem solving Professional ethics Management theory Economic constraints Quantitative skills Diverse workforce Public policy* Globalization Role of public service* Concentration expertise Notes: * Sample size 104, except for those with an asterisk, which had 62. Interpretation of these results is a bit challenging, though note especially that many students have had a high impression of their skill levels on entering the program, with between 8-38% of incoming students reporting that they need no improvement, and generally under a third of students reporting that they need significant improvement. As shown in Table 6 below, though, on exiting the program a large portion of students report improvement, or significant improvement in skill levels. The program has performed especially well in terms of understanding the role of public service, local governance, and their concentration area of expertise; and has also been strong in professional ethics and economics. Page 4 of 12

5 Table 6 Student skill self-assessment: post-test (Capstone PAD6066) Student skills were Communication: -- writing speaking listening Local governance Problem solving Professional ethics Management theory Economic constraints Quantitative skills Diverse workforce Public policy Spring 12 Globalization Role of public service Spring 12 Concentration expertise Class Unchanged Unchanged (prior exp.) Improved Improved significantly Page 5 of 12

6 As indicated, the survey was also carried out on alumni in Respondents were asked to assess their skill development as a result of the MPA program. Alumni self-assessment results are presented in Table 7. Table 7 Alumni perceptions of skill assessment Unchanged Alumni skills were Improved Unchanged (prior exp.) Improved significantly Communication: -- writing speaking listening Local governance Problem solving Professional ethics Management theory Economic constraints Quantitative skills Diverse workforce Public policy Globalization Role of public service Concentration expertise Change in life satisfaction 9.8 n/a The results are strong across the board. Only in an understanding of globalization (which has not been part of the curriculum, or mission prior to 2011), diversity, and perhaps communication skills, have there been historical weaknesses. Yet at least 75%, and as many as 95% of respondents reported improvement on all items. 3. Student Civic Literacy quiz The Student Civics quiz is based on a national survey, which allows for comparison outside the program. The survey consists of a number of Constitutional, Historical and Economic questions. The survey seeks to test whether the program has provided a broad-based understanding of the context of governance. Results are presented in Table 5. The process used in this evaluation is also a pre-test (PAD 6060 Intro) and post-test (PAD 6066 Capstone). Students complete this quiz anonymously, and are not graded on it. As of Spring 2014, we have applied the quiz to six Intro classes (n = 96), and four Capstone classes (n = 107). Prior to this semester, results have been mixed. While UNF-MPA Capstone students have consistently outperformed incoming students, these differences have not been statistically significant. As we gathered more data it was expected that this would change, and we reached that point in Spring As can be seen, the number of correct answers (out of 33 possible) is Page 6 of 12

7 1.4 higher for Capstone students, at a 95% level of statistical confidence (p =.039). In the subcategories, the Constitutional questions are especially higher (by 6.1%, p =.015) for Capstone students, while the other two categories have lower levels of statistical significance. Table 8 Student civic literacy survey Constitutional History Economic Civic literacy (% correct) #/33 Intro (n = 96) Capstone (n = 107) (p =.015) (p =.169) (p =.107) (p =.039) US undergrads (n = 587) 18.8 Intro (n = 96) 21.7 (p =.000) US grads (n = 412) 21.1 Capstone (n = 107) 23.1 (p =.000) The UNF MPA program also consistently outperforms the broader population, both on entering and leaving the MPA program. Though a fairly quixotic assessment tool, the logic behind it has been to test to see if, as a result of time in the program, students develop a richer understanding of the broader context in which public managers operate, despite our not specifically teaching either these topics, or the specific questions in the quiz. 4. Public Management essay The third assessment exercise applied by the UNF-MPA program is a public management writing exercise, also applied as a pre-test (PAD 6060) and post-test (PAD 6066). To date the essay has been taken by three Introductory class, and the four Capstone classes reported. Students write an essay on similar articles in each exercise, and are assessed on the following Learning Outcomes: Discipline knowledge. Demonstrate command of the public management literature. Critical thinking. Demonstrate the ability to make a logical, coherent, balanced argument. Research. Demonstrate the ability to gather evidence. Writing. Demonstrate written communication competence. Each learning outcome is scored for each student on a 0-3 scale, as shown: 3 Mastery, above normal requirements for professional work. 2 At a level appropriate for professional work. 1 Insufficient for professional work, requires minor improvement. 0 Insufficient for professional work, requires considerable improvement. Table 9-11 present the results. Table 9 presents results by the percentage of students who fall in each rubric category for each learning outcome. With one exception (the research score for the Spring 2011 Capstone), each Capstone class outscored the combined scores for the Introductory classes. Page 7 of 12

8 Discipline knowledge Critical analysis Research Writing Table 9 Writing exercise, by learning outcomes Class Insufficient major Insufficient minor Prof. Adequate Prof. Mastery Mean Score Intro Caps-Spg Caps-Spg Caps-Spg Caps-Spg Intro Caps-Spg Caps-Spg Caps-Spg Caps-Spg Intro Caps-Spg Caps-Spg Caps-Spg Caps-Spg Intro Caps-Spg Caps-Spg Caps-Spg Caps-Spg Table 10 combines these scores for each individual student. So while Table 9 combines the scores of students for each learning outcome, Table 10 indicates the combined (the four Learning Outcomes averaged) performance of individual students. Each cell presents the number of students whose average score on the four Learning Outcomes was in each of these ranges. The four categories are broadly equivalent to the four categories used in Table 9: an overall score of 0-1 equates to major insufficiency, 1.25 to 1.75 minor insufficiency, 2 to 2.25 professional sufficiency, and 2.5+ professional mastery. Table 10 Writing exercise, by average student score on all four items. Class Class Frequency distribution, average score Class Mean Median Intro- Caps-Spg 11 Caps-Spg Caps-Spg Caps-Spg The Table demonstrates strong improvement, with about half of incoming students having professionally insufficient writing skills, and this percentage dropping to under 20% by the Capstone class. That about 20% of Capstone students are unable to write to a standard adequate Page 8 of 12

9 for a professional environment, though, suggests that this is an area that the program needs to work on. Finally, Table 11 presents tests of statistical significance between the Introductory (n = 57) and Capstone (n = 102) students. All scores are higher for the Capstone class. Except for the quality of Research in these short papers, the other four differences are all statistically significant: weak in the case of Writing, but very strong in Discipline Knowledge, Critical Analysis, and Combined. Table 11 Intro and Capstone compared, writing exercise Assessment Class Mean Probability item score Discipline Intro knowledge Capstone 2.33 Critical analysis Intro Capstone 2.39 Research Intro Capstone 2.22 Writing Intro Capstone 2.36 Combined Intro Capstone Quantitative analysis exercise The UNF-MPA program adopted a fourth major assessment item in Spring : an assessment of student quantitative analysis skills. While Table 6 shows that nearly all Capstone students have subjectively indicated that their quantitative skills were improved, this exercise applies an objective test to that assessment. Students were presented with raw data on education and income in Florida and Indiana counties. Students were also presented with SPSS correlation and regression output using that data, asked to interpret this data, and present a table with education and income figures for a small number of northeast Florida counties. Students were assessed on their knowledge of quantitative analysis, their critical analytical skills, and their table presentation. Results are presented in Table 10. Table 12 Quantitative analysis exercise, by Capstone learning outcomes (%, and on 0-3 scale for mean score) Statistical interpretation (n = 25) 2014 (n = 33) Critical analysis 2014 Table presentation 2014 Combined 2014 Insufficient major Insufficient minor Prof. Adequate Prof. Mastery Mean Score Page 9 of 12

10 6. Concentration assessments. The program also administers a Local Government competence essay, and a Concentration Competence essay. These exercises have been carried out (and reported) for Spring, and are in progress for Spring However while initial results are good, numbers are small, so take the Concentration assessment results in Table 13 as preliminary. Table 13 Concentration assessment exercise, by Spring -14 Capstone competency Insufficient major Insufficient minor Prof. Adequate Prof. Mastery Nonprofit concentration 1 Critical policy thinking (#) Mean score Management literature (#) Combined Local government concentration Critical policy thinking (#) Management literature (#) Combined Health admin concentration Critical policy thinking (#) Management literature (#) Combined Public Policy concentration 2 Public policy literature (#) Critical policy thinking (#) (#) Combined Conflict transformation Applied ethics These two independent programs are concentration options for MPA students. To date no student has completed either. Notes: 1 Double graded. The others have not been to date Spring 2014 only, as this concentration was added in Fall 2012, and the first graduate occurred in Fall. Page 10 of 12

11 7. Program student numbers and academic qualifications Recent student numbers have returned to levels seen prior to the recession. The total number of enrolled students (see Table 14) in the program has varied from over the past couple of years. Enrollments (total class enrollments) have also returned to their pre-recessionary levels. This is shown in Table 15, below. For the past few years, we have offered 20 classes a year: eight each in fall and Spring, and four in the summer, though are offering seven in Spring We expect a decrease in student numbers in the short term, as a too large (37 student) Spring 2014 Capstone class graduates over the next year. Table 14 Enrolled student numbers MPA Fall Spring NP Cert Fall Spring Post-bac Fall Spring Total Fall Spring Table 15 Total class enrollments, and graduates Class totals a Fall Spring b Summer Total Graduates Total Notes: a b Total enrollments in all courses, as opposed to the Table 3 number of individual students in the program. Application periods are set by the Graduate College, and refer to the semester of admission Page 11 of 12

12 Table 16 presents recent data on enquiries, applications, and admission decisions. Numbers appear to be holding steady. Spring 2011 Sum 2011 Table 16 Enquiries, applications and admissions Fall Spring Sum Fall Spring Sum Fall Enquiries a MPA NP cert MPA App ns b Admit Provisional Deny Others c Enroll d NP cert -- App ns Admit Deny Spring 2014 Notes: a Enquiries from September-December for Fall; January-April for Spring; May-August for Summer. The numbers represent individual prospects who contacted the program during the period in question. b Application periods are set by the Graduate College, and refer to the semester of admission c d Readmitted students, and successful post-baccalaureate students. This is data from the Graduate School. Table 17 presents applicant academic qualifications. Note that MPA admission requirements are a 3.0 GPA, and combined GRE score of 1000 (on the old measurement system). There has been a slight drop in these credentials, as a result of the admission of 2-3 students with low academic credentials, but strong work experience or successful completion of classes as Post- Baccalaureate students. Spg 2011 Table 17 Applicant academic qualifications Fall Spg Sum Fall Sum 2011 Spg Sum Fall Spg 2014 MPA Admit GPA GRE Provisional GPA GRE Deny GPA GRE NP cert. Admit GPA n/a n/a Deny GPA Page 12 of 12