MPA Program Assessment Report Summer 2015

Transcription

1 MPA Program Assessment Report Summer 2015 Introduction: This was the second full year for doing learning outcomes assessment based on the 2009 NASPAA accreditation standards and conducting our exit interviews using an on-line survey. For the first time, the program assessed all five of NASPAA s core competency areas. In addition, the program tested a new cultural competency component, to be fielded in its final form in the Fall 2015 semester. Finally, one MPA student (as part of a directed study in program evaluation) surveyed her peers on student support services. That data is also included in this report. Exit Interview Results Student Assessment of Learning: Students believe they have learned quite a bit from our program. They were asked to evaluate what they knew relative to our learning outcomes at the beginning of our program and what they believe they could do now. Table 1 presents the contrast. Table 1: Self-Assessment of Learning Before Joining the Program At the Conclusion of the Program Properly identify human, organizational, and resource issues in a case or situation and develop an appropriate course of action Describe how public policy decisions are made Identify and explain the political, social, and economic contexts in which public policy decisions are made Collect and analyze data essential to understanding and addressing problems Properly identify the legal and/or constitutional issues in a case or situation and develop a course of action that is legal and constitutional Properly identify a public administrator s responsibility to elected officials and the general public in a case or situation and develop a course of action or solution that considers these issues Effectively communicate in writing Effective communicate orally in a public presentation Work in a team In two open-ended questions, students were asked to identify additional skills or knowledge gained from the program, along with what they wished they had learned.

2 Additional skills learned included the ability to better spot weaknesses in work, skills in public budgeting, and learning about the political process. The wished for list included the injection of more real-world experiences into coursework. Student Assessment of Advising Feedback on advising was generally very positive, as shown in Table 2 below: Table 2: Student Evaluation of Advisor Strongly disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Willing to meet with me at a time and place that was convenient Helpful and courteous Helped me develop a course schedule that fit my needs Responded to my questions in an accurate and timely manner Recommending the Program to Others Eight of 9 students indicated they had recommended our program to people they know. The one who did not criticized overuse of powerpoint by some professors (unnamed). Student Assessment of Quality of Instruction Students were asked to assess the quality of instruction for both core and elective courses. Table 3 and 4 present the results. Table 3 (next page) presents the results for core courses.

3 Table 3: Student Assessment of Quality of Instruction, Core Courses Strongly disagree Disagree Neither Agree nor Disagree Agree Strongly Agree PADM 6000, Survey of Public Administration PADM 6050, Constitutional and Administrative Law PADM 6100, Organizational Theory and Behavior PADM 6200, Human Resource Management PADM 6300, Public Budgeting PADM 6600, Quantitative Methods PADM 6650, Public Policy Analysis PADM 6750, Program Evaluation Table 4 (next page) presents the results for elective courses. The numbers vary since students make different choices as to which electives to take. Overall, however, the feedback was very positive and clearly comparable to the results for core courses.

4 Table 4: Student Assessment of Quality of Instruction, Elective Courses Strongly disagree Disagree Neither Agree nor Disagree Agree Strongly agree Did not take the course PADM 6020, GIS PADM 6030, Grant Writing & Administration PADM 6250, Introduction to Urban Planning PADM 6301, Financial Management for Nonprofits PADM 6302, Nonprofit Management PADM 6350, Emergency Management PADM 6351, Introduction to Homeland Security PADM 6352, The Unconventional Threat PADM 6550, Human Services Administration PADM 6700, Urban Government Administration Learning Outcomes Assessment This academic year, we were scheduled to conduct assessments in five courses: PADM 6100, 6050, 6600, 6650, and This academic year, all of NASPAA s 5 core

5 competencies were evaluated. Table 5 summarizes the results of learning outcomes assessment this academic year. Table 5: Results of Learning Outcomes Assessment in Courses Course PADM 6100 NASPAA Competency To lead and manage in public governance Learning Outcome Students will be able to compare and contrast a given pair of leadership theories Assessment Method & Standard 80% of students will correctly answer exam question Results 11 of 12 students scored 4 or better on the rubric, meeting the standard. PADM 6600 PADM 6050 To analyze, synthesize, think critically, solve problems and make decisions To articulate and apply a public service perspective Students will be able to analyze data Students will be able to properly identify legal and Constitutional issues Students shall score one unit better on the post-test than the pre-test 80% of students will correctly identify legal and Constitutional issues Four of six students scored 2-4 units better on the post-test than on the pre-test; one remained the same and the sixth student s performance declined by 3 points. However, the class average was one unit better on the post-test than the pre-test. All students scored 16 or more on the case assignment; all students met the standard. PADM 6650 PADM 6900 To participate in and contribute to the policy making process To communicate and interact with a diverse and changing workforce and citizenry Students will be able to describe the policy-making process, including its actors Students will be able to demonstrate the ability to effectively communicate and interact with others in a team project, negotiation, or an internship 80% of students will correctly answer exam question 80% of students will demonstrate the ability to effectively communicate and interact with others by achieving a score of 30 or higher on peer evaluations or site supervisor evaluations This learning outcome was NOT met; of the 10 papers evaluated only 6 exceeded the standard grade of 70% All students exceeded the standard of 30 on site supervisor evaluations; the class average was

6 This round of learning outcome assessments was both instructive and helpful. As a result of assessment, we learned that we need to focus more attention on the structure and content of PADM This is the second year in which students in this course did not successfully meet the expected learning outcomes standard. Since two different instructors were involved, we can conclude it was not the instructor. Both instructors did, however, use the same text and general approach to the class. Clearly, this needs significant work. Cultural Competency Component In the academic year, the faculty completed a review of our curriculum to see how we were covering cultural competency (diversity) issues. Carrizales (2010) recommends curriculum components include 1) knowledge of local and national demographics, societal disparities, and policy and legal issues surrounding diversity; 2) a self-reflection component that includes societal biases; 3) a skills-based component that focuses on communication skills; and 4) a community-based component that includes internships and other experiential learning offerings. Our review found that we were missing the self-reflection component. As a result, the MPA Director contacted Dr. Lorraine Evans from the Office of Diversity and Inclusion who put together an on-line component that could be used with PADM 6000 as the self-reflection component and would introduce students to cultural competency. That component was tested with the PADM 6100 class in the Spring 2015 term. The results were overwhelmingly positive. The pre/post test was adapted from the Clinical Cultural Competency Questionnaire (CCCQ) developed by Robert C. Like, MD, MS, Professor and Director of the Center for Healthy Families and Cultural Diversity, Department of Family Medicine, UMDNJ-Robert Wood Johnson Medical School. The 14-question survey addressed perceived overall self-efficacy as well as knowledge, confidence, skills and self-awareness, all key constructs in cultural competency, as well as overall perspectives on the topic and general demographics. The results are depicted in Table 6 on the next page.

7 Table 6: Pre/Post Test Score Comparison I consider myself to be culturally competent I feel confident in my ability to utilize communication skills to improve service encounters with people of diverse backgrounds (e.g. different ages, genders, ethnicities, language skills, and socioeconomic statuses). It is important for public administration professionals to receive training in cultural competency. *=p<.05, **= p<.01 N = 9 Pre-test mean Post-test mean Change * ** In addition to the pre-test/post-test, students also wrote a self-reflective essay focused on cultural learning, unconscious bias and promoting an inclusive workplace. The goal was to encourage critical reflection on the topic and required students to connect the material to real world problems. It was a pass/fail assignment and all students successfully completed the essay. The cultural competency component will be given in conjunction with PADM 6000, Survey of Public Administration, beginning with the Fall 2015 semester and continue as a component in that class. Student Services As part of a directed study version of PADM 6750, Program Evaluation, Sophie Spencer developed, conducted, and analyzed a survey of her fellow students to assess student satisfaction with campus services. A summary of broad results is presented in Table 7 below; a more in-depth discussion of bookstore, library, and career services is provided after Table 7 on the next page.

8 Table 7: Summary of Results from Student Services Survey Item Perceptions of campus safety Classroom quality Food service Class registration Student financial accounts Admissions process Financial aid Advising Internships Finding(s) 13 of 15 respondents said they felt safe or very safe on campus 11 of 15 respondents were satisfied or very satisfied with the quality of classrooms 3 of 14 respondents were dissatisfied or very dissatisfied with food services; another 6 were neutral All 15 respondents reported registering for classes on-line; 13 of them were satisfied or very satisfied with that process All 15 respondents know how to access their student financial account on-line; 12 of them were satisfied or very satisfied with that service. 10 of 15 students were either satisfied or very satisfied with the admissions process While students had a high degree of familiarity with where to apply, satisfaction rates were lower. Of the 12 students who had used financial aid, only half (6) were either satisfied or very satisfied. All respondents knew who their advisor was, and 8 of 15 claimed to communicate frequently or very frequently with her. 12 of 15 were either satisfied (4) or very satisfied (8) with the advisor. All respondents knew who to contact for information about an internship. The 8 who had had an internship all said they were very satisfied (6) or satisfied (2) with the experience itself, the help getting the internship, and the advising process for the internship as Bookstore: 14 out of 15 respondents knew where the bookstore was, but half of them claimed they never went there. When asked to rate the bookstore experience, the students were fairly satisfied with service but highly dissatisfied with price. 10 of 15 respondents said they were dissatisfied (3) or very dissatisfied (7) with the cost of books. Library: Interestingly, respondents were just as likely to use the library on campus as to use its on-line services, as illustrated in Table 8. Not at all Sometimes Frequently Very Frequently Physically going to the library Online access Online librarian assistance On campus librarian assistance

9 Satisfaction with library services was VERY high, with higher satisfaction rates reported for on campus experiences. For example, 13 of 15 students were either very satisfied or satisfied with on campus services, but just 8 of 15 rated online services that way. Career Services: While most students know what Career Services does and where it is located, the overwhelming majority are not using those services. As an example, the most frequently used service was resume development, and just 5 of 15 students had used that service. Marketing In the academic year, the program faculty adopted a marketing plan with an ambitious goal of reaching an enrollment of 45 by August 1, Although the action steps listed in the marketing plan were pursued, and in most cases completed, this goal will not be met. Actions Taken: Program director and/or faculty attended 3 graduate fairs, and TGS carried MPA promotional materials to an additional 3 graduate fairs. One of those fairs was the one held in Atlanta, for the HBCUs in that area. Promotional materials were acquired. A new brochure was developed and printed in the summer of 2014, and additional materials were ordered in May of this year to assist with next year s recruitment. A special handout was developed and taken to graduate fairs to highlight to relative low cost of our program compared to others. The website was overhauled to make it more current and attractive Approval was received (in concept) for two new additions a nonprofit certificate and a BA to MPA option that could help attract new students to the program Additional Steps Needed: Separation of the MPA Director duties from the Center for Public Service will take place July 1 of this year. This will allow the new MPA Director to focus exclusively on the program and student recruitment. The current MPA Director has been moved back into teaching required courses at the undergraduate level. This will help ensure undergraduates in our department have contact with MPA faculty on a more regular basis and should help recruitment. Cultural Competency (Diversity) The specific status of each of the goals set in our plan is attached to this report. The significant accomplishments this year included:

10 Raising $15,000 for a new scholarship to assist students in defraying the cost of completing an MPA degree Improvements in the advising process to include mandatory in-person advising for new students and follow-up on their performance in the first semester of classes Creation of a self-awareness component (discussed earlier under learning outcomes assessment) Overall Conclusions Graduating students believe they learned quite a bit from the program, were quite satisfied with advising, and the overwhelming majority are recommending the program to others. Once again, students in PADM 6650 did not achieve the desired learning outcomes. It is recommended that this course receive a complete overhaul (new text and approach) to ensure students learn what we would like them to learn. Generally, students are satisfied with services on the campus. However, students could benefit from more contact with career services. The faculty should discuss was to integrate career services into courses, perhaps as brief presentations, to help sell students on the importance of using those services. Recruiting of students needs significant attention. Steps taken this year should lay the groundwork for better results next year; however, the new director will need to make this a priority.

11 Target for recruitment undergraduate institutions and programs popular with minority groups. Cultural Competency Status Strategy Tactic(s) Responsible Party Status as of Spring 2015 Attend at least two graduate Ribando/Hammond Complete school/career fairs sponsored by institutions or consortia with large populations of minority students. Highlight the low cost of our program compared to other institutions Locate donors to fund scholarships for MPA students 1. Develop a handout comparing our tuition to others for use at recruiting fairs. 2. Put information about our costs prominently on our website. 1. Community Advisory Board member(s) will lead effort to raise funds for Ralph H. Walker scholarship 2. Use awareness event for the Center for Public Service to raise funds for the scholarship (10/24) Ribando Complete Clayton & Ross/Ribando Complete; $15,000 raised. Development of process/policy for scholarship underway. Expected award for Fall 2015 Make in-person advising mandatory for students in their first semester Improve communication between instructors of first semester courses and the MPA Director Schedule in-person advising with each new student in the first semester to assess career goals, experience and courses needed to support those goals. Director to contact instructors to explicitly ask for feedback on the performance of new students in their first semester at least one. Ribando Complete Ribando/Hammond/Miller Complete

12 Strategy Tactic(s) Responsible Party Status as of Spring 2015 Raise awareness of our new status as part of an R1 institution Participate as board members, committee members, or active volunteers in national and regional professional associations All On-going Actively recruit a diverse part-time faculty Add a self-reflection component to the curriculum to promote cultural competency Use community connections to search for part-time faculty from underrepresented groups to teach in the program or serve as guest speakers Adapt the existing on-line component on cultural competency designed for healthcare professionals to public administration. Incorporate that into the first semester experience for all new students and make completion mandatory for graduation. Ribando Ribando List of guest speakers provided to Dr. Hammond for use in 6650; no current need for additional part-time faculty Incorporated into PADM 6100 as pilot; fully operational in Fall 2015