Program Review Guidelines Undergraduate Programs

Transcription

1 Program Review Guidelines Undergraduate Programs Purpose Program review is an opportunity for programs without discipline specific accreditation or approval to engage in: Synthesis and evaluation of the degree to which the program meets intended operational and student learning outcomes; Strategic planning to guide continuous improvement of the program; Peer review to gather constructive feedback from the field regarding the quality of the program. It is coordinated by the every 7 years for programs without periodic, discipline specific accreditation or approval. New programs undergo program review in their 4 th year in preparation for post-implementation review by the Alabama Commission on Higher Education and every 7 years thereafter. All programs that do not have periodic, discipline specific accreditation or approval are required to participate. The process generally takes about 12 months to complete; however, the maximum amount of time allotted to complete the self-study is three semesters (not including summer). Appendix A includes a timeline for completing the self-study. The department s and program s annual assessment and data reporting should be sufficient to identify the 3-year trend data needed to complete the self-study. Outcomes and conclusions identified in the self-study and/or the external review should be used to develop subsequent annual assessment plans for programs, departments, and colleges/schools, as appropriate. The Program Review process includes: A self-study of the program; A review of the self-study by a qualified external reviewer; A review and approval of the self-study by the Dean, AVP for Institutional Effectiveness, and the SVPAA/VPHS, as appropriate; 6/17/2014 Page 1 of 14 Draft

2 Contents of the Self-Study: 1) Title sheet indicating the program s official name and CIP code 2) Signature page (see Appendix B) 3) Self-Study Summary Summary of significant conclusions drawn from the program review process to include identification of the program s strengths, areas in need of improvement, and proposed goals and actions that will inform future program development. Summary may be presented in the form of a SWOT analysis to include strengths, weaknesses, opportunities, and threats. Data/evidence, which should be detailed in the full document, must be used to justify conclusions. 4) Program Overview Description of the program to include the mission statement, a brief history of the program, its role within the College/School and University, and how it supports the University s mission and strategic plan. A degree plan that lists all courses and the semester in which these courses are typically offered. A curriculum map that reflects connections between student learning outcomes, assessment processes, and required courses. 5) Program Resources (see Appendix C for required data/evidence). To what extent is student enrollment adequate and/or appropriate to support the program? To what extent are faculty qualifications and numbers sufficient to support the program? To what extent is the program utilizing technology to improve course/program delivery and/or enhance instruction? To what extent are facilities, space, technology, and other resources sufficient to support the program (Data provided, in part, from USA facilities survey)? To what extent do students and faculty have access to the library resources and services that support the program? Also, please identify what resources and services may be lacking, if any. Other important resource areas as identified by program faculty 6) Student Learning Outcomes Assessment: Please consider data you have collected for each of the last three years on the program s student learning outcomes individually and as a unit (see Appendix D for required data/evidence and additional questions to prompt faculty inquiry). What can faculty conclude about the quality of student learning? What can faculty conclude about the ability of their students to progress through and graduate from the program? What can faculty conclude about the quality of their teaching? 6/17/2014 Page 2 of 14 Draft

3 What can faculty conclude about the preparedness of their graduates to enter the workforce and/or continue to post-baccalaureate education? 7) Approval from Chair and/or Dean and Sr. VPAA indicating self-study is ready for external review (see Appendix B). 8) External review Minimum qualifications for the external reviewer Certified reviewer from a professional association or a distinguished colleague in the discipline with regard to teaching and scholarship. Must not have a conflict of interest (e.g., current/former USA faculty, family member or business partner with any current departmental faculty, etc.). Process for choosing an external reviewer Prior to identifying potential reviewers, the Dean and Department chair should meet to determine whether the external reviewer will conduct a paper review only or also conduct a site visit. The decision to host a site visit is made by the Dean. At this meeting, the Dean and Department Chair also should clarify expectations for qualifications of the reviewer, expectations for the self-study, and expectations for the site-visit (if applicable). Department chair and/or faculty identify three potential reviewers and submit curriculum vitas for the three individuals to the Dean. The Dean makes the selection and notifies the department chair. External reviewer report (See Appendix F for rubric) While there is no standard format for the external reviewer s report, he or she should use the rubric to evaluate the self-study and use the 1 st and 2 nd level headers as an outline to structure the report. Additional areas of review and evaluation (beyond those covered by the rubric) are encouraged and welcome as appropriate. 9) External reviewer report from the Council on Undergraduate Research (optional and strongly encouraged). Visit for more information or contact USA s Undergraduate Research Program at ucur@southalabama.edu. 10) Required Appendices Chair and program coordinator/director curriculum vitae Course syllabi Student/Program handbooks, if applicable Required tables (see Appendices C and D) Other data, evidence, or other supporting documentation as applicable or needed to justify conclusions about programs effectiveness. 6/17/2014 Page 3 of 14 Draft

4 Appendix A Month/Date July August (last half) September (first half) September (last half) October March April May August (last half) September (last business day) October (last business day) November (last business day) December (last business day) Activity Offices of Institutional Effectiveness (IE) and Institutional Research (IR) prepare program review materials and program specific data. Program coordinator and/or chair meets with (IE) to review the program review process and schedule. Chair and/or program coordinator meets with faculty and staff. Chair and/or program coordinator submits list of potential external reviewers to the Dean for approval. Dean submits list of approved reviewers back to chair and/or program coordinator. Chair and/or program coordinator secures external reviewer. Program staff and faculty prepare self-study. Program staff and faculty submit the self-study for an internal review and to the Dean, AVPIE, and Sr. VPAA for preliminary approval. Chair and/or program coordinator sends the report to the external reviewer. No Campus Visit With Campus Visit External reviewer s report submitted to Dean; Review external reviewer s Prepare for campus visit. comments with the dean, chair/program coordinator, and faculty. Revise assessment plan, update results and develop actions based on findings of the self-study and the external reviewer s recommendations. Report in TracDat. Submit the final self-study to Dean, AVPIE, and Sr. VPAA for approval. Send a copy of the approved Self-study to OIE Upload report into TracDat as supporting documentation External reviewer campus visit completed. External reviewer s report submitted to Dean; Review external reviewer s comments with the dean, chair/program coordinator, and faculty. Revise assessment plan, update results and develop actions based on findings of the self-study and the external reviewer s recommendations. Report in TracDat. Submit the final self-study to Dean, AVPIE, and Sr. VPAA for approval. Send a copy of the approved Self-study to OIE Upload report into TracDat as supporting documentation 6/17/2014 Page 4 of 14 Draft

5 APPENDIX B THE UNIVERSITY OF SOUTH ALABAMA Academic Program Review for Undergraduate Programs Self-Study Approval Signature Page Department: Program: Title Date Printed Name Signature Department Chair Dean Assoc. VP for Institutional Effectiveness Sr. VP for Academic Affairs VP for Health Sciences 6/17/2014 Page 5 of 14 Draft

6 Appendix C List of Required Data/Evidence for Program Evaluation Program Review of Undergraduate Programs Table 1: Program level enrollment trends for the last five years for both full-time and part-time students. Table 2: Program graduation trends for last five years (headcount) (also ACHE needed for post-implementation review). Table 3: Program new student enrollments for the last five years (headcount only) (also ACHE needed for post-implementation review). Table 4: List of identified peer programs (University, degree, and program name) Table 5: List of current program faculty, teaching load, courses most often taught, degrees, recent scholarly/creative activity (last three years), and usage of highimpact practices (see Appendix E). Table 6: Participation in high impact practices. Table may include high-impact practices and numbers of students participating or identification of courses in which the practice was used (last two years). For courses, please list course name/number and the semester in which it was taught. Special Note: For programs submitting ACHE post-implementation reviews in the year following program review, two other conditions also must be addressed; however, data for these two conditions is provided by the Office of Institutional Research. 1. IPEDS degree completion 2. IPEDS projected degree completion 6/17/2014 Page 6 of 14 Draft

7 Appendix D List of Required Data/Evidence for Student Learning Outcomes Program Review of Undergraduate Programs All required data/evidence is available either 1) through your annual TracDat reports or 2) from the Office of Institutional Research or. Program faculty should feel free to include other data in addition to the required elements to support their decisions and/or conclusions in the self-study. Table1: Program level aggregation of student mastery of student learning outcomes for the last three years (also needed for ACHE post-implementation review). Table 2: Program level aggregation of student retention (1 and 2 year) and completion (4 and 6 year) rates. For the last three cohorts, as applicable. Table 3: Program level aggregation of student evaluation of teaching for the last three years. Table 4: Program level aggregation of post- graduation outcomes assessment for the last three years. This may be include one or more of the following (but not limited to) exit surveys, alumni surveys, employment rates, graduate school acceptance rates, etc. (also needed for ACHE post-implementation review). 6/17/2014 Page 7 of 14 Draft

8 Appendix D (cont.) Questions to prompt inquiry about Student Learning Outcomes Assessment Student Learning Outcomes Assessment: Please consider data you have collected for each of the last three years on the program s student learning outcomes individually and as a unit. What can faculty conclude about the quality of student learning? Questions to prompt inquiry: Is the standard for achievement set at an appropriate level? Is the standard understood and agreed upon by all faculty? Are there student learning outcomes where students perform particularly poorly or particularly well? Are there student learning outcomes where the standard for achievement should be raised or lowered? Where data are available, how do student compare to students in other USA programs or nationally? What can faculty conclude about the ability of their students to progress through and graduate from the program? Questions to prompt inquiry: Do students exhibit difficulty progressing through the major at any point? If no, what are the things that faculty do that support student progression? If yes, at what points and/or why? Do students exhibit difficulty completing the program? If no, what are the things that faculty do that support student completion? If yes, at what points and/or why? What can faculty conclude about the quality of their teaching? Questions to prompt inquiry: If there are areas of student learning identified as needing improvement (improving outcomes or raising standards), is modification/improvement of instruction identified as a possible means to improvement? Have there been efforts made to support and/or improve the quality of faculty instruction? Did program faculty implement any high impact practices and what was their effect, if any, on student learning outcomes, faculty or student engagement, etc. (see Appendix E for a non-exhaustive list of high impact practices)? What can faculty conclude about the preparedness of their graduates to enter the workforce and/or continued post-baccalaureate education? Questions to prompt inquiry: Consider the program s student learning outcomes and how they translate to knowledge, skills, and/or abilities that make your students attractive to potential employers and/or graduate school? 6/17/2014 Page 8 of 14 Draft

9 Appendix E List of High Impact Practices for student success/engagement in undergraduate programs (not exhaustive) Service-Learning Undergraduate Research (UCUR) Learning Communities First-Year Experience Capstone Experiences Team-Based Learning Intrusive advisement Study Abroad Use of Supplemental Instruction 6/17/2014 Page 9 of 14 Draft

10 Appendix F Program Review (undergraduate programs): Self-Study Evaluation Rubric (under development) 1 Beginning 2 Developing 3 Good 4 Exemplary Student Learning Outcomes* (SACS ) Clarity and Specificity of SLO s No outcomes stated. Orientation of SLO s No outcomes stated in student-centered terms. Outcomes present, but with imprecise verbs (e.g., know), vague description of content/skill/or attitudinal domain, and non-specificity of whom should be assessed (e.g., students ) Some outcomes stated in student-centered terms. Curriculum mapping of SLO s to courses No activities/courses Activities/courses listed but link listed. to outcomes is absent. Outcomes generally contain precise verbs (e.g, rich description of the content/skill/or attitudinal domain, such as those in Bloom s Taxonomy, and specification of whom should be assessed (e.g., graduating seniors in the Biology B.A. program ) Most outcomes stated in student-centered terms. Most outcomes have classes and/or activities linked to them. All outcomes stated with clarity and specificity including precise verbs, such as those in Bloom s Taxonomy, rich description of the content/skill/or attitudinal domain, and specification of whom should be assessed (e.g., graduating seniors in the Biology B.A. program ) All outcomes stated in studentcentered terms (i.e., what a student should know, think, or do). All outcomes have classes and/or activities linked to them. Relationship between measures and SLO s Seemingly no At a superficial level, it appears relationship between the content assessed by the outcomes and measures matches the measures. outcomes, but no explanation is provided. Types of measures No measures indicated Most outcomes assessed primarily via indirect (e.g., surveys) measures. General detail about how outcomes relate to measures is provided. For example, the faculty wrote items to match the outcomes, or the instrument was selected because its general description appeared to match our outcomes. Most outcomes assessed primarily via direct measures. Detail is provided regarding objectiveto-measure match. Specific items on the test are linked to outcomes. The match is affirmed by faculty subject experts (e.g., through backwards translation). All outcomes assessed using at least one direct measure (e.g., tests, essays). 6/17/2014 Page 1 of 14 Draft

11 Appendix F 1 Beginning 2 Developing 3 Good 4 Exemplary Student Learning Outcomes* (SACS ) Specification of desired results No a priori desired results for outcomes Statement of desired result (e.g., student growth, comparison to previous year s data, comparison to faculty standards, performance vs. a criterion), but no specificity (e.g., students will grow; students will perform better than last year) Data collection and research design integrity No information is provided about data collection process or data not collected. Limited information is provided about data collection such as who and how many took the assessment, but not enough to judge the veracity of the process (e.g., thirty-five seniors took the test). Additional validity evidence (as appropriate to the measure being used) No additional psychometric properties provided. Reliability estimates (e.g., internal consistency, test-retest, inter-rater) provided for most scores, although reliability tends to be poor (<.60). Or, author states how efforts have been made to improve reliability (e.g., raters were trained or rubric). Desired result specified. (e.g., our students will gain ½ standard deviation from junior to senior year; our students will score above a faculty-determined standard). Gathering baseline data is acceptable for this rating. Enough information is provided to understand the data collection process, such as a description of the sample, testing protocol, testing conditions, and student motivation. Nevertheless, several methodological flaws are evident such as unrepresentative sampling, inappropriate testing conditions, one rater for ratings, or mismatch with specification of desired results. Reliability estimates provided for most scores, most scores are marginal or better (>.60). Desired result specified and justified (e.g., Last year the typical student scored 20 points on measure x. The current cohort underwent more extensive coursework in the area, so we hope that the average student scores 22 points or better.) The data collection process is clearly explained and is appropriate to the specification of desired results (e.g., representative sampling, adequate motivation, two or more trained raters for performance assessment, pre-post design to measure gain, cutoff defended for performance vs. a criterion) Reliability estimates provided, most scores are marginal or better (>.60). Plus, other evidence given such as relationship of scores to other variables and how such relationship strengthens or weakens argument for validity of test scores. 1 Beginning 2 Developing 3 Good 4 Exemplary 6/17/2014 Page 2 of 14 Draft

12 Appendix F Presentation of results No results presented History of results No results presented Interpretation of results No interpretation attempted Dissemination of results No evidence of communication Results are present, but it is unclear how they relate to the outcomes or the desired results for the outcomes. Only current year s results provided Interpretation attempted, but the interpretation does not refer back to the outcomes or desired results of outcomes. Or, the interpretations are clearly not supported by the methodology and/or results. Information provided to limited number of faculty or communication process unclear. Student Learning Outcomes* (SACS ) Results are present, and they directly relate to the outcomes and the desired results for outcomes but presentation is sloppy or difficult to follow. Statistical analysis may or may not be present. Past iteration(s) of results (e.g., last year s) provided for some assessments in addition to current year s. Interpretations of results seem to be reasonable inferences given the outcomes, desired results of outcomes, and methodology. Information provided to all faculty, mode and details of communication clear. Results are present, and they directly relate to outcomes and the desired results for outcomes, are clearly presented, and were derived by appropriate statistical analyses. Past iteration(s) of results (e.g., last year s) provided for majority of assessments in addition to current year s. Interpretations of results seem to be reasonable given the outcomes, desired results of outcomes, and methodology. Plus multiple faculty interpreted results (not just one person). And, interpretation includes how classes/activities might have affected results. Information provided to all faculty, mode and details of communication clear. In addition, information shared with others such as advisory committees, other stakeholders, or to conference attendees. 1 Beginning 2 Developing 3 Good 4 Exemplary 6/17/2014 Page 3 of 14 Draft

13 Appendix F Student Learning Outcomes* (SACS ) Use of results for improvement/modification (Student learning and development) No mention of any improvements. Examples of improvements documented but the link between them and the assessment findings is not clear. Use of results for improvement (assessment process) No mention of how this Some critical evaluation of past iteration of assessment and current assessment, is improved from past including acknowledgement of administrations. flaws, but no evidence of improving upon past assessment or making plans to improve assessment in future iterations. Examples of improvements (or plans to improve) documented and directly related to findings of assessment. However, the improvements lack specificity. Critical evaluation of past and current assessment, including acknowledgement of flaws; Plus evidence of some moderate revision, or general plans for improvement of assessment process. Examples of improvements (or plans to improve) documented and directly related to findings of assessment. These improvements are very specific (e.g., approximate dates of implementation and where in curriculum they will occur.) Critical evaluation of past and current assessment, including acknowledgement of flaws; both present improvements and intended improvements are provided; for both, specific details are given. Either present improvements or intended improvements must encompass a major revision. *Nearly all of the rubric for Student Learning Outcomes is borrowed from James Madison University s Assessment Progress Template (APT) Evaluation Rubric 2013 by Keston H. Fulcher, Donna L. Sundre, & Javarro A. Russell. 6/17/2014 Page 4 of 14 Draft

14 Appendix F 1 Beginning 2 Developing 3 Good 4--Exemplary Program Resources Use of technology for instruction (3.4.12) TBD TBD TBD TBD Use of high-impact practices ( ) No students or faculty are actively engaged in high-impact practices at least once annually. Very few students or very few faculty are engaged in highimpact practices at least once annually. Approximately half of the students and faculty are engaged in high-impact practices at least once annually. Faculty qualifications (3.7.1) TBD TBD TBD TBD Faculty resources (3.5.4) At least 25% of course hours in the major are taught by faculty holding an appropriate terminal degree. At least 50% of course hours in the major are taught by faculty holding an appropriate terminal degree. At least 75% of course hours in the major are taught by faculty holding an appropriate terminal degree. Adequacy of facilities (space) (3.11.3) TBD TBD TBD TBD Adequacy of facilities (technology) (3.4.12) TBD TBD TBD TBD 1 Beginning 2 Developing 3 Good 4--Exemplary Overall Evaluation ( ) Major findings and/or conclusions TBD TBD TBD TBD Use of review process to drive planning TBD TBD TBD TBD Nearly all of the students and faculty are engaged in highimpact practices at least once annually. All or nearly all course hours in the major are taught by faculty holding an appropriate terminal degree. 6/17/2014 Page 5 of 14 Draft