Systematic Program Review Procedures Antelope Valley College

Transcription

1 Systematic Program Review Procedures Antelope Valley College Approved May 17, 2012 Updated to include report templates for

2 Table of Contents I. Program Review Process Page 3 Background Information Guiding Principles Integration with Outcomes Assessment and Strategic Planning Self-study Process Publication and Discussion of the Findings II. Comprehensive Program Review (Four-Year Review) Page 6 Initial Planning and Training Time Frame Template for Comprehensive Program Review Report Peer Review Guidelines and Procedures III. Annual Program Review Update Page 13 Time Frame Template for Annual Update Program Review Report Peer Review Guidelines and Procedures IV. Appendix A - ACCJC: Characteristics of Evidence Page 18 2

3 I. Program Review Process Background Information Program review at Antelope Valley College is part of the overall planning and institutional effectiveness process. It is collaborative goal setting and assessment designed to assist faculty, staff, and administrators with continually refining and improving program practices resulting in appropriate improvements in outcomes and student achievement. Every program and service in the college goes through an annual systematic self-evaluation process and institutionally based peer review. The primary purpose of program review is to provide ongoing assessment of institutional advancements in effectiveness. The information gathered during this process provides a basis for informed decision-making by faculty, staff, and administration regarding the future of the institution and resource allocations by the Strategic Planning and Budget Council (SPBC). Budget requests to SPBC will only be reviewed if supported by an upto-date program review report. Accreditation standards reflect the importance of program review in sustaining continuous quality improvement. Evaluations, Planning, and Improvement, a theme of accreditation emphasized throughout the standards, requires colleges to maintain an ongoing and systematic cycle of evaluation. This planning cycle is accomplished in part by the program review process. Program review permits the college to evaluate, set goals, distribute resources, implement goals, and then re-evaluate. The program review self-study process aids programs in: Strengthening programs through self-improvement and self-determination. Generating continuous and ongoing dialogue about how outcomes and student achievement can be enhanced through program and service improvements. Evaluating their contribution to achieving the college mission, vision and Institutional Learning Outcomes (ILOs). Helping programs clarify goals and align them with the Educational Master Plan (EMP). There are two rotating processes in program review with each resulting in the preparation and submission of a self-study report. They are: Comprehensive Program Review, occurring at least every four years, focuses broadly on the program and improvement of outcomes and student achievement. It should have real impact on program effectiveness and resource allocation decisions. The comprehensive self-study report builds the foundation for the annual update report. The Program Review Coordinator in consultation with the Academic Senate President and Executive Vice President of Academic Affairs and Student Services shall establish the frequency and order in which programs will be reviewed. 3

4 Annual Program Review Update asks each program to provide a report on outcome and achievement assessment activities for the prior academic year and update its needs for resources. Updates are done annually between comprehensive reviews. Both the comprehensive and annual update reports will be examined by members of the Program Review Committee. Committee membership reflects broad district representation. Guiding Principles Program review is utilized by the program to assess the effectiveness of existing programs and create plans to improve outcomes and advance student achievement. It is not a process for evaluating individual performance or for program discontinuance. The following principles should guide the program review process: Collegiality: The process should be a faculty/staff-driven, collaborative process guided by a spirit of open and honest inquiry. Relevance: The process should analyze appropriate data to support resource requests and answer important questions for the program. Practicality: The self-study report should be as short as possible (30 pages maximum for the comprehensive report and 10 pages maximum for the annual update report, not including attachments). Effectiveness: The process should result in a clear sense of direction and accomplishment for participants. To facilitate the implementation of plans and the accomplishment of goals, resource needs identified through the selfstudy process will be linked with ILOs. Timeliness: A self-study report must be completed and submitted to the Program Review Coordinator annually by October 31. Integration with Outcomes Assessment and Strategic Planning Outcomes Assessment Programs are asked in the program review process to analyze assessment results for student learning outcomes (SLOs), program learning outcomes (PLOs) and operational outcomes (OOs). They are also asked to evaluate the impact of resource allocations on improving outcomes. Strategic Planning The information gathered during the program review process supports informed planning by faculty, staff, and administration regarding the future of the institution and resource allocations. Ongoing program assessment creates a culture of informed decision-making and quality improvement. Self-study reports will be made available to the entire community online. The following flow chart shows the overall linkages between program review and other aspects of district planning. 4

5 Strategic Planning and Budget Council (SPBC) Documents Self-study Process As in the accreditation model, the preparation of the self-study report is viewed as the primary vehicle by which the faculty and staff directly involved in the program under review assume responsibility for assessing and improving program practices. The written self-study report shall include an analysis of outcomes and student achievement. The program under review identifies specific goals and plans for improvement that support the college mission, vision, ILOs, and Educational Master Plan. Any findings that support a modification of the Educational Master Plan should also be included. The self-study team shall rely upon quantitative and qualitative data as a basis for preparing and writing the self-study report. Institutional data shall be analyzed. The program may include and analyze data compiled by the program or other sources to aid in evaluating the program. The source of data used to support trends or conclusions shall be identified in the self-study report. If the self-study team takes exception to data provided by the Department of Institutional Effectiveness, Research, and Planning, this should be noted in the self-study report. While the dean is ultimately responsible for the presentation of the self-study report, all full time faculty and staff involved in the program shall collaborate and participate in the preparation and review of the document. Adjunct faculty and part time staff will be 5

6 encouraged to participate in the process. Each program should select program representatives who will provide organizational leadership for completion of the review. The effort to reach consensus by the dean and staff of the program, especially on major goals and objectives, should be viewed as a central feature of the evaluation and planning process. When consensus on a specific issue cannot be reached the self-study report may reflect the differences or the dissenting staff member(s) may attach an addendum to the section in question. Programs conducting self-studies will have support from the Department of Institutional Effectiveness, Research, and Planning. Standardized longitudinal and student progress data will be provided on the web link. Additional data not included in the standardized data set will be provided upon request. The Accrediting Commission for Community and Junior Colleges (ACCJC) provides specific characteristics of evidence that programs should consider while gathering appropriate data for their respective areas. See Appendix A - ACCJC: Characteristics of Evidence on page 18. II. Comprehensive Program Review (Four-Year Review) Initial Planning and Training As many faculty and staff members of the program as possible should attend the training meeting. The training meeting will: Remind programs about ACCJC accreditation standards, the program review process, assessment of outcomes and student achievement, and the role of program review in strategic planning and budgeting. Provide an opportunity for the self-study team to dialogue with the Department of Institutional Effectiveness, Research, and Planning about data provided and needed. Provide a discussion of the long range planning analysis from the Educational Master Plan. Provide a forum for discussion of any concerns about the process. Assist programs in the development of a project plan with timelines and assignment of responsibilities, if needed. Remind programs that the Program Review Coordinator will be available throughout the process to discuss progress on the review and any areas where the self-study team needs assistance. Time Frame Spring/Summer Initial planning and training for programs doing comprehensive reviews. 6

7 Discuss program review procedures and guidelines within the program. Develop detailed task list and timeline which allows adequate time to write, review, discuss, and revise drafts. Gather any additional information the program deems necessary. Revise and update Course Outlines of Record (CORs) and submit to the Academic Policies and Procedures Committee (AP&P), if needed. Review and analyze the data provided on the Web link and received from the Department of Institutional Effectiveness, Research, and Planning. Review and analyze the data collected by the program. Request additional data, if needed, from the Department of Institutional Effectiveness, Research, and Planning at least 30 days before needed to analyze the data. Engage in dialogue about outcomes and their connection to college ILOs and strategic goals. Review prior program review reports. Responses that remain accurate, relevant and appropriate to this report may be resubmitted. Prepare draft report resulting from dialogue within the program. September - October Review and finalize the comprehensive self-study report. The dean presents the comprehensive self-study report in October to the Program Review Coordinator for peer review by the Program Review Committee. November - March The self-study report is read by members of the Program Review Committee. They recommend either full or conditional approval of the comprehensive report. Conditional approval will require the program self-study team to make specified revisions to the self-study report to gain full approval. Program Review Coordinator will forward an electronic copy of the peer review report to the dean. The dean will disseminate the peer review report to the staff within the program being reviewed. Upon full approval of the comprehensive self-study report by the Program Review Committee, the Program Review Coordinator posts the self-study report and the peer review report to the Academic Senate web page and places a copy of each report in the AVC Library. March Program Review Coordinator notifies the district president, vice presidents, and SPBC co-chairs about completion of comprehensive self-study and peer review reports. SPBC utilizes information gathered during the self-study process as a basis for informed decision-making by faculty, staff, and administration regarding the future of the institution and resource allocations. Template for Comprehensive Program Review Report 7

8 Comprehensive Program Review Report Due October 31 Please provide the following information. Respond NA to questions which are not applicable to your division/discipline/area. The self-study reports of all divisions/areas will include responses to Parts 1-7. Self-study reports of academic divisions will include a division overview in Part 1 and analysis of each discipline in Parts 2-7. Questions with an asterisk (*) were addressed in last year s program review report. The question numbers do not correspond with the numbers in last year s report. Comprehensive Program Review Self-Study Report Division/Area Name Year Part 1 - Division or Area Overview 1.1 Briefly describe how the division or area contributes to the district mission. 1.2 Place an X by each Institutional Learning Outcome (ILO) supported by the division or area. Analyze diverse perspectives from a variety of disciplines and experiences that contribute to the development of self-awareness. Value and apply lifelong learning skills required for employment, basic skills, transfer education, and personal development. Demonstrate a breadth of knowledge and experiences from the humanities, social and behavioral sciences, arts, natural sciences, and mathematics. Solve problems using oral and written communication, critical thinking and listening skills, planning and decision-making skills, information literacy, and a variety of technologies. Demonstrate good citizenship and teamwork through respect, tolerance, cultural awareness, and the role of diversity in modern society. Identify career opportunities that contribute to the economic well-being of the community. 1.3 After completing Parts 2-7, prepare a one page summary of the division/area. Interpret the significance of the findings. Note successes in supporting district strategic goals and where improvements are needed. 1.4 Name of person leading this review 1.5 Names of all participants in this review 8

9 Part 2 - Data Analysis and Use The following data is provided on the Program Review website. Additional data is available from the Department of Institutional Research and Effectiveness (DIERP). Longitudinal data District headcount and FTES Division headcount and FTES Discipline headcount and FTES Number of sections offered by location/distance education PT/FT faculty ratio by LHE Efficiency (measured as FTES/FTEF) Data about student progress Student achievement: success, retention, and term to term persistence Progression through remedial courses Program completion Degree/certificate completion rate Transfer rates to 4-year institutions Licensure exam results Job placement/post training All division/areas will complete Parts 2-7. In academic divisions Parts 2-7 will be completed by each discipline; please identify the discipline: 2.1 Please review the five year headcount and FTES enrollment data provided on the web link. Comment on trends and how they affect your program.* 2.2 Report and analyze program/area data showing the quantity of services provided over the past four years (e.g. number of students served, books sold, employees hired, acreage maintained). 2.3 Please review the five year data on sections offered, faculty ratios, and efficiency data provided on the web link. Comment on trends and how they affect your program. 2.4 Using the discipline student success data provided by web link, please comment on any similarities or differences between race, gender, location, and modality groups in meeting the Institutional Standard of 68% for student success (students earning grades of A, B, C, Pass, or Credit). Identify what actions are planned to address trends and achievement gaps in the current academic year.* 2.5 Analyze and summarize trends in student progression through basic skills courses, if applicable. 9

10 2.6 List degrees and certificates currently offered in the discipline. Analyze how resource adjustments or other changes during the past four years have impacted degree and certificate completion rates. 2.7 Using the data provided by web link, please comment on transfer rates to four-year institutions, license exam results, and job placement/post testing. If applicable, cite examples of using additional resources (e.g. human, facilities/physical, technology, financial, professional development) or making other changes during the past four years that have resulted in improvements in transfer rates to four-year institutions, license exam results, and job placement/post testing. 2.8 Career Technical Education (CTE) programs: Review the labor market data on the California Employment Development Department website for jobs related to your discipline. Comment on the occupational projections for employment in your discipline for the next two years. Comment on how the projections affect your planning. Part 3 Outcome Analysis and Use 3.1 Analyze changes in student learning outcome (SLO) and program learning outcome (PLO) assessment findings over the past five years. Cite examples of using data during that time as the basis for resource allocation (e.g. human, facilities/physical, technology, financial, professional development) or making other changes that resulted in or correlate with improved learning outcome findings over the past four years.* 3.2 Analyze changes in operational outcomes (OO) findings over the past five years. Cite examples of using data during that time as the basis for resource allocation (e.g. human, facilities/physical, technology, financial, professional development) or making other changes that resulted in or correlate with improved OO findings over the past four years.* Part 4 - Stakeholder Assessment 4.1 Assess how well the program serves the needs of the students, district, and community. Support statements with findings from student, employee, and/or community surveys. Include feedback from other sources if relevant (e.g. advisory committees, employers in the community, universities, scores on licensure exams, job placement). Part 5 - Goals and Objectives 5.1 Review the goals identified in your most recent comprehensive self-study report and last year s annual report. Indicate which have been completed and which have been eliminated.* 5.2 List discipline/area goals and objectives related to improving outcome findings and/or the success of the various learner populations in completing courses, certificates, degrees, and transfer requirements. Discipline/area goals must be guided by 10

11 district Strategic Goals and Plan Summaries in the Educational Master Plan (EMP). They must be supported by an outcome action plan, data analysis, national or professional standards, and/or a requirement or guideline from an outside agency (e.g. legislation, Chancellor s Office, accrediting body, professional board). Consider curriculum, instruction, assessments, program services, operations, collaborations, scheduling, location, technology, etc.* Current (up to three years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting action plan, data analysis, or other documentation Objectives: Significant steps or actions needed to achieve the goal Near Term (three to five years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting action plan, data analysis, or other documentation Objectives: Significant steps or actions needed to achieve the goal Long Term (five to ten years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting action plan, data analysis, or other documentation Objectives: Significant steps or actions needed to achieve the goal 5.3 List discipline/area goals and objectives directly related to advancing Strategic Goals. Discipline/area goals must be guided by district Strategic Goals and Plan Summaries in the Educational Master Plan (EMP). They must be supported by data analysis or other documentation. Current (up to three years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting data analysis or other documentation Objectives: Significant steps or actions needed to achieve the goal Near Term (three to five years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting data analysis or other documentation Objectives: Significant steps or actions needed to achieve the goal 11

12 Long Term (five to ten years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting data analysis or other documentation Objectives: Significant steps or actions needed to achieve the goal Part 6 - Resource Needs Identify significant resource needs that should be addressed currently (up to three years), near term (three to five years), and long term (five to ten years). If there may be safety issues, enrollment consequences, or other important concerns if a resource is not provided please make this known.* 6.1 List needed human resources. List titles in priority order. Identify which discipline/area goal(s) guides this need. 6.2 List needed technology resources in priority order. Identify which discipline/area goal(s) guides this need. 6.3 List facilities/physical resources (remodels, renovations, or new) needed to provide a safe and appropriate student learning and/or work environment. List needs in priority order. Identify which discipline/area goal(s) guides this need. 6.4 List needed professional development resources in priority order. Identify which discipline/area goal(s) guides this need. 6.5 List any other needed resources in priority order. Identify which discipline/area goal(s) guides this need. Part 7 - Recommendations and Comments 7.1 List recommended changes to the Educational Master Plan to: Address external issues or mandates such as legislation, industry, and professional standards, etc. Respond to outcome findings. Reflect changes in technology, methodology, and/or disciplines. Address student achievement gaps and/or meet other student needs. 7.2 What changes in the program review process would improve institutional effectiveness or make the results more helpful to the program? Peer Review Guidelines and Procedures for the Comprehensive Report Comprehensive self-study reports will be read by at least two members of the Program Review Committee. Members of the Program Review Committee will be guided by the 12

13 program review process and objectives and examine the program self-study report using a rubric. They will then: Note their general reactions to and perceptions of the report. Identify areas that require clarification, verification, or additional information. Recommend either full or conditional approval of comprehensive and annual update reports. Conditional approval will require the program self-study team to make specified revisions to the report to gain full approval. If the two readers cannot reach agreement on rubric scoring and/or comments to the self-study team, the Program Review Coordinator or designee will read the report and collaborate with the original two readers to determine the majority view. The end result will be a brief peer review report confirming the self-study report is complete and the program documented data use to inform goals and other decisions intended to improve outcomes and/or student achievement. The Program Review Coordinator shall serve as a resource person for Program Review Committee members, monitor the peer review process, and serve as a liaison between the committee members, the Department of Institutional Effectiveness, Research, and Planning, Academic Senate, and Executive Vice President of Academic Affairs and Student Services. Dissemination of the Peer Review Report Upon full approval of the comprehensive self-study report, the Program Review Coordinator will forward the peer review report to the dean of the program under review. The dean will disseminate the report to the staff within the program being reviewed. The Program Review Coordinator will post the self-study report and the peer review report to the Program Review website and place a copy of each report in the AVC Library. III. Annual Program Review Update The Annual Program Review Update is conducted by all programs in each of the three years between comprehensive self-study program reviews. It focuses on select areas of the Comprehensive Program Review, including data analysis, updating goals, and identifying needs. It should be as brief as possible, finalized, and posted in WEAVE by October 31. Time Frame Spring/Summer Discuss program review procedures and guidelines within the program. Develop detailed task list and timeline which allows adequate time to write, review, discuss, and revise drafts. Gather any additional information the program deems necessary. Revise and update Course Outlines of Record (CORs) and submit to the Academic Policies and Procedures Committee (AP&P), if needed. 13

14 Review and analyze the data provided on the web link and received from the Department of Institutional Effectiveness, Research, and Planning. Review and analyze the data collected by the program. Request additional data, if needed, from the Department of Institutional Effectiveness, Research, and Planning at least 30 days before needed to analyze the data. Engage in dialogue about outcomes and their connection to college ILOs and strategic goals. Review prior program review reports. Responses that remain accurate, relevant and appropriate to this report may be resubmitted. Prepare draft report resulting from dialogue within the program. September - October Review and finalize the comprehensive self-study report. Post final version in WEAVE by October 31. November - March The annual update self-study report is read by members of the Program Review Committee. They recommend either full or conditional approval of the annual update report. Conditional approval will require the program self-study team to make specified revisions to the self-study report to gain full approval. Program Review Coordinator will forward an electronic copy of the peer review report to the dean. The dean will disseminate the peer review report to the staff within the program being reviewed. Upon full approval of the annual update self-study report by the Program Review Committee, the Program Review Coordinator posts the self-study report and the peer review report to the Academic Senate Web page and places a copy of each report in the AVC Library. March Program Review Coordinator notifies the district president, vice presidents, and SPBC co-chairs about completion of annual update and peer review reports. SPBC utilizes information gathered during the self-study process as a basis for informed decision-making by faculty, staff, and administration regarding the future of the institution and resource allocations. The Program Review Coordinator will be available to discuss progress on the review and any areas where the self-study team needs assistance. Template for Annual Update Program Review Report Annual Update Program Review Report Due October 31 Annual Update Program Review Report 14

15 Please provide the following information. Respond NA to questions which are not applicable to your discipline/program/area. Questions with an asterisk (*) were addressed in last year s program review report. The question numbers may not correspond with the numbers in last year s report. 1. Discipline/Area Name 2. Year 3. Name of person leading this review 4. Names of all district participants in this review Data/Outcome Analysis and Use 5. Please review the five year headcount and FTES enrollment data provided by web link. Comment on trends and how they affect your program.* 6. Using the student achievement data provided by web link, please comment on any similarities or differences in success, retention, and persistence between race, gender, location, and modality groups. Please comment on all three (success, persistence, and retention). Note where improvement is needed to meet the Institutional Standard of 68% for student success (students earning grades of A, B, C, Pass, or Credit). Identify what actions are planned to address trends and achievement gaps in success, retention, and/or persistence in the current academic year.* 7. Analyze changes in student achievement and achievement gaps over the past five years. Cite examples of using data during that time as the basis for resource allocation (e.g. human, facilities/physical, technology, financial, professional development) or making other changes that resulted in or correlate with improved student achievement.* 8. Career Technical Education (CTE) programs: Review the labor market data on the California Employment Development Department website for jobs related to your discipline. Comment on the occupational projections for employment in your discipline for the next two years. Comment on how the projections affect your planning Analyze changes in SLO, PLO, and/or OO assessment findings over the past five years. Cite examples of using data during that time as the basis for resource allocation (e.g. human, facilities/physical, technology, financial, professional development) or making other changes that resulted in or correlate with improved SLO, PLO, and/or OO findings this past year.* Goals and Objectives 15

16 10. Review the goals identified in your most recent comprehensive self-study report and last year s annual report. Indicate which have been completed and which have been eliminated.* 11. List current (up to three years) discipline/area goals, and objectives related to improving outcome findings and/or the success of the various learner populations in completing courses, certificates, degrees and transfer requirements. Discipline/area goals must be guided by district Strategic Goals and Plan Summaries in the Educational Master Plan (EMP). They must be supported by an outcome action plan, data analysis, national or professional standards, and/or a requirement or guideline from an outside agency (e.g. legislation, Chancellor s Office, accrediting body, professional board). Consider curriculum, instruction, assessments, program services, operations, collaborations, scheduling, location, technology, etc.* Current (up to three years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting action plan, data analysis, or other documentation Objectives: Significant steps or actions needed to achieve the goal 12. List current (up to three years) discipline/area goals, and objectives directly related to advancing Strategic Goals. Discipline/area goals must be guided by district Strategic Goals and Plan Summaries in the Educational Master Plan (EMP). They must be supported by data analysis or other documentation. Current (up to three years) Goal: A specific target Guided by district Strategic Goal(s) # Guided by Plan Summary in EMP Supporting action plan, data analysis, or other documentation Objectives: Significant steps or actions needed to achieve the goal Resource Needs 13. Identify significant resource needs which should be currently addressed (up to three years). If there may be safety issues, enrollment consequences, or other important concerns if a resource is not provided please make this known.* List needed human resources. List titles in priority order. Identify which discipline/area goal(s) guides this need. List needed technology resources in priority order. Identify which discipline/area goal(s) guides this need. List facilities/physical resources (remodels, renovations or new) needed to provide a safe and appropriate student learning and/or work environment. List needs in priority order. Identify which discipline/area goal(s) guides this need. List needed professional development resources in priority order. Identify which discipline/area goal(s) guides this need. List any other needed resources in priority order. Identify which discipline/area goal(s) guides this need. 16

17 Peer Review Guidelines and Procedures for the Annual Update Report Annual Update reports will be read by at least two members of the Program Review Committee. Members of the Program Review Committee will be guided by the program review process and objectives and examine the program self-study report using a rubric. They will then: Note their general reactions to and perceptions of the report. Identify areas that require clarification, verification, or additional information. Recommend either full or conditional approval of comprehensive and annual update reports. Conditional approval will require the program self-study team to make specified revisions to the report to gain full approval. If the two readers cannot reach agreement on rubric scoring and/or comments to the self-study team, the Program Review Coordinator or designee will read the report and collaborate with the original two readers to determine the majority view. The end result will be a brief peer review report confirming the self-study report is complete and the program documented data use to inform goals and other decisions intended to improve outcomes and/or student achievement. The Program Review Coordinator shall serve as a resource person for Program Review Committee members, monitor the peer review process, and serve as a liaison between the committee members, the Department of Institutional Effectiveness, Research, and Planning, Academic Senate, and Executive Vice President of Academic Affairs and Student Services. Dissemination of the Peer Review Report Upon full approval of the comprehensive self-study report, the Program Review Coordinator will forward the peer review report to the dean of the program under review. The dean will disseminate the report to the staff within the program being reviewed. The Program Review Coordinator will post the self-study report and the peer review report to the Program Review website and place a copy of each report in the AVC Library. 17

18 Appendix A ACCJC: Characteristics of Evidence Evidence is the data upon which a judgment or conclusion may be based. As such, it is presented in answer to questions that have been deliberately posed because an institution regards them as important. Evidence tells all stakeholders that an institution has investigated its questions and knows something about itself it knows what it achieves. For evidence to be useful, it must have undergone analysis and reflection by the college community. The dialogue required for analysis and reflection is an integral part of the capacity an institution has for using the evidence it has accrued to make improvements. Good evidence, then, is obviously related to the questions the college has investigated and it can be replicated, making it reliable. Good evidence is representative of what is, not just an isolated case, and it is information upon which an institution can take action to improve. It is, in short, relevant, verifiable, representative, and actionable. Evidence on Student Achievement and Student Learning The evidence the institution presents should be about achievements (student movement through the institution) and should include the following: Student preparedness for college, including presormance on placement tests and/or placement, Student training, needs, including local employment training needs, transfer education needs, basic skills needs, etc., Course completion data, Retention of students from term to term, Student progression to the next course/next level of course, Student program (major) completion, Student graduation rates, Student transfer rates to four-year institutions, Student job placement rates, Student scores on licensure exams. The evidence the institution presents should also be about student learning outcomes (mastery of the knowledge, skills, abilities, competencies, attitudes, beliefs, opinions, and Values at the course, program, and degree levels in the context of each college s mission and population) and should include data on the following: Development and dissemination of student leaning outcomes, Samples of student work/performance (recitals, projects, capstone courses, etc.), Summary data on measured student leaning outcomes, Measurement and analysis of student attainment of student learning outcomes used as part of the institution s self-evaluation and planning processes, Improvement of the teaching/learning process as a result of the above analysis. 18

19 Self-study should be only one phase of on-going institutional evaluation, and an evaluating team should be able to see how the institution develops and uses evidence of effectiveness as part of its ongoing evaluative processes. Institutions should gather and use both qualitative and quantitative evidence, and often just use indirect as well as direct measures to assess institutional effectiveness. Good evidence used in evaluations has the following characteristics: It is intentional, and a dialogue about its meaning and relevance has taken place. It is purposeful, designed to answer questions the institution has raised It has been interpreted and reflected upon, not just offered up in its raw or unanalyzed form. It is integrated and presented in a context of other information about the institution that creates a holistic view of the institution or program. It is cumulative and is corroborated by multiple sources of data. It is coherent and sound enough to provide guidance for improvement. It is important to note that evidence per se does not lead to confirmations of value and quality. Rather, the members of the college community must arrive at the decisions about the value and quality through active judgments. The purpose of good evidence is to encourage informed institutional dialogue that engages the college community and leads to improvements of its processes, procedures, policies, relationships, ultimately with the effect of improving student learning. Good evidence should provide the means for institutions or evaluators to make sound judgments about quality and future directions, but at the same time, it will probably stimulate further inquiry about institutional quality. 19