KILGORE COLLEGE Planning and Evaluation Handbook October 2008 Research and Institutional Effectiveness Office 1100 Broadway Kilgore, Texas 75662-3204 903-983-8620
Table of Contents Planning and Evaluation at Kilgore College... 1 History of Planning and Evaluation at KC... 1 Current Process... 2 Training/Education... 4 Outcomes and Assessment... 5 Annual Improvement Plan (AIP) Process... 6 Kilgore College s Mission and Vision Statements... 6 AIP Flowcharts: Annual Improvement Plan Flowchart... 7 Instructional Units Annual Improvement Plan Flowchart... 8 Administrative & Educational Support Service Units AIP Flowchat... 9 Planning and Evaluation Templates: Unit Outcomes Template... 10 Directions for Completing Unit Outcomes... 11 Annual Improvement Plan Template... 12 Directions for Completing Annual Improvement Plan... 13 Annual Improvement Plan Summary Template... 14 Directions for Completing Annual Improvement Plan Summary... 15
Planning and Evaluation at Kilgore College Kilgore College (KC) engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; and (3) demonstrate that the institution is effectively accomplishing its mission. KC has a history of identifying expected outcomes, assessing the extent to which it achieves these outcomes, and collecting evidence of improvement based on analysis of the results in units throughout the institution. All educational, administrative, and educational support units, including those who are engaged in community service, participate in the process. The formal initiative, started in 1993, has continued to develop and mature as a vital part of the College s overall institutional effectiveness process. History of Planning and Evaluation at KC The institution has been actively engaged in documented institutional research and effectiveness since 1993 when the president of the College established an office of institutional research, staffed by an institutional data specialist. Later that same year, an institutional effectiveness committee was formed to develop a systematic process for evaluating the extent to which the College was effectively accomplishing its mission. This committee was later renamed the Research and Institutional Effectiveness (RIE) Committee to more accurately describe the scope of the committee as well as the importance of institutional research in the planning and evaluation process of the College. At the macro level, strategic planning has been an ongoing process for over 10 years. During this time, the College has implemented three distinct strategic plans, each addressing needs of the College at their respective times. The first plan spanned from 1997 to 2001 and included issues related to student success, workforce development, student diversity, technology, employee benefits, faculty development, financial resources, and marketing. Realizing that assistance was needed in identifying and addressing the needs of its community and in marketing its programs and services, the College began working with the Clarus Corporation in 2002 to conduct a comprehensive environmental scan of multiple constituencies in its service area. The College invested $80,000 in this project, which resulted in 93 recommendations and a five-year work plan. This strategic plan ran from 2002 to 2007. As action on the Clarus strategic plan was being completed, the College began developing its current plan, which is being implemented for 2008 2011. This plan is aligned with the structure of the state-wide strategic plan from the Texas Higher Education Coordinating Board (THECB), Closing the Gaps. Additional information on the Clarus Strategic Plan and the current strategic plan is given later in this narrative. The Institutional Effectiveness Committee developed a strategic planning matrix on which each unit of the College would document its expected outcomes, assessment procedures and criteria, assessment results, and improvement plans. The original three-page format was condensed to two pages and then to a one-page institutional effectiveness matrix in 1995. The Committee was later renamed the Research and Institutional Effectiveness (RIE) Committee to more accurately describe the scope of the Committee as well as the importance of institutional research in the planning and evaluation process of the College. Through its RIE Committee, the College evaluates its own process of improvement planning. Evaluating the way that all instructional, administrative and educational support units had documented their planning and evaluation activity during the 2000-2001 cycle, the Committee 1
found that, while all individual units were involved in identifying desired outcomes and in collecting and analyzing data, 44% of the units had written some objectives that were not measurable or had reported that required data were missing or inconclusive. Furthermore, only 51% of all units had implemented viable improvement plans. The analysis of this evaluation led the Committee to determine that the current annual improvement planning matrix was not adequately providing the format to allow for the implementation of viable improvement planning for all units. To assist employees with their understanding of the components of the planning and evaluation cycle, the RIE Committee replaced the one-page institutional effectiveness matrix with the Annual Improvement Plan (AIP) in 2002. This new format allows for units to document not only their assessment and improvement planning initiatives but also to document the evaluation of the effectiveness of strategies in the improvement plan. This led to the development and implementation of the AIP and AIP Summary documents for the 2003-2004 cycle. These instruments have proven to be much more effective in documenting actual improvement initiatives. Using these templates, 100% of all units develop plans for improvement. Improvement results are documented by each unit on the AIP Summary, which is submitted to the unit head s supervisor and to the director of RIE after the improvement plan strategy is conducted and results are evaluated. Improvement results are compiled in an annual RIE report. The College compiled its first IE/AIP Summary Report in 1997 and has continued to do so since that time. Current Process As depicted in the model below, planning and evaluation at KC follow a cyclical process of defining desired outcomes, implementing strategies, gathering and analyzing data, creating and implementing improvement plans, and evaluating the effectiveness of plans (data gathering and analysis). The mission and core values of the institution are at the center of all improvement initiatives, which are implemented at all levels. 2
Although not exhaustive of all KC planning events, the following timetable includes most of the regularly scheduled action items in its ongoing process of planning and evaluation: Date August September October November December January February March April May Summer Action Dean of institutional planning meets with unit heads during opening week activities. Instructional and educational support services units review unit purpose, analyze research data, and formulate improvement plans. Student evaluation of instructor/course for non-credit courses (4 th qtr). Graduating Student Survey (summer graduates). Annual Improvement Plan (AIP) summaries due for fiscal year improvement plans. Units submit AIP to unit s administrative supervisor via e-mail attachment. Research and institutional planning personnel available for comment/feedback. RIE committee meets to review previous year s planning process and research data. AIP presentation meetings begin for all instructional, administrative, and educational support services units. Revised AIPs emailed to unit supervisor and to director of RIE. AIP presentation meetings continue for all instructional, administrative, and educational support services units. Revised AIPs emailed to unit supervisor and to director of RIE. Program reviews conducted for instructional programs/departments. Student evaluation of instructor/course for non-credit courses (1 st quarter) Graduating Student Survey (fall graduates). Student evaluation of instructor/course for semester credit courses (KC Instructor Evaluation). AIP summary due for fall improvement plans. Board review of mission, goals, outcomes. Student Needs Survey and/or Student Satisfaction Survey. Student evaluation of instructor/course for non-credit courses (2 nd quarter). Educational support units planning meetings. Evaluation of faculty (biannual) and staff (annual) completed. Budget process begins. Evaluation by employees of KC services (Institutional Effectiveness Survey). Budget requests due to Executive Council (EC) members; budget adjustments. Student evaluation of instructor/course for selected semester credit courses and/or instructors (KC Instructor Evaluation). AIP summaries due for fall/spring improvement plans. Budget adjustments continue. Graduating Student Survey (spring graduates). Budget submitted to Board for approval. New Student Orientation Surveys. Review of selected educational support services (3-year rotation). Student evaluation of instructor/course for non-credit courses (3 rd quarter). EC reviews/analyzes data from Balanced Scorecard and other institutional data sources. Administrative units formulate improvement plans. At the institution level, the strategic planning process is consistent with the ongoing and multilevel process of data collection and analysis at the unit level. Using trend and peer comparison data from the THECB Accountability System, KC Fact Book, and input from their respective component areas of the College, members of the EC construct the strategic plan, identifying goals, objectives, strategies, specific desired outcomes, the office responsible for the strategy, the timeline of implementation, and the budget and other resources needed to accomplish the strategy. Of particular importance, is the Balanced Scorecard, a document noting institutional indicators related to the KC mission and core values. Goals and targets on the Scorecard were developed by the RIE Committee and the EC. 3
Currently led by the director of RIE, the RIE office compiles numerous reports which units throughout the institution use for improvement initiatives. AIPs and the strategic plan include documentation of the data which were used to support the need for improvement. Information given in both the AIP and the AIP Summary require units to supply research-based data to support their plan and to support the success of their plan. Improvement results are documented by each unit on the AIP Summary, which is submitted to the unit head s supervisor and to the director of RIE after the improvement plan strategy is conducted and results are evaluated. Improvement results are compiled in an annual RIE report. The director of RIE compiles improvement results from the strategic plan on an ongoing basis. They are reviewed by the EC and are reported to the Board of Trustees in February. The College s planning and evaluation processes are ongoing. The College has a history of engagement in a cyclical process of identifying expected outcomes, conducting assessment of student learning and administrative and educational support processes, identifying issues needing improvement, implementing improvement plans, and checking for actual improvement in the process. Processes are integrated, with all planning activity related to the mission and core values of the institution. These processes are also institution-wide, allowing for participation and input from appropriate faculty and staff at the course, unit (department/program), and institution levels. Planning and evaluation at KC is data-informed, institutional research being an active and essential element of the process. As part of its cyclical process, the College systematically reviews its mission, goals, and outcomes. Through these ongoing, integrated, and institution-wide planning and evaluation processes, KC is actively engaged in improving its quality at all levels; through these collective processes, the College demonstrates that it is effectively accomplishing its mission. 2008/2009 RIE Committee Roster: Robin Huskey, Director of RIE; D Wayne Shaw, Automotive Technology instructor; Staci Martin, Registrar; Julian Redfearn, Business & Management instructor; Lara Pauley, Early Childhood instructor; Charleen Worsham, Director of Instructional Technology & Distance Education; Becky Johnson, English instructor and Chair of Language Development Dept.; Dean Kinney, Criminal Justice instructor; Bindy Tice, Trio Director; Debbie Williams, Biology instructor; Lucy Carroll, Office Professional Instructor; Nina Verheyden, Mathematics instructor; Carolyn Williams, Bookstore Manager; Mike Jenkins, VP of Student Development; Mike Turpin, Dean of Institutional Planning (chair). Training /Education As part of KC s institutional effectiveness process, each educational, administrative, and educational support unit, including those which engage in community service, participates in planning and evaluation processes designed to measure the extent to which each unit meets its own purpose, supports the mission of the College, and engages in continuous improvement. All unit heads meet at least annually to discuss processes for identifying outcomes, success standards, measurement procedures, and timelines. At the fall meeting of unit heads, the dean of institutional planning reviews the planning and evaluation cycle and addresses any topics of concern. Unit head meetings include a review of the College s mission and core values, directions for completing unit/student learning outcomes, and directions for completing the AIP and the AIP Summary. Process flow and timelines are also discussed. During the week of registration for the fall and spring semesters, the College also designates time for instructional unit leaders to meet with their respective units regarding planning and evaluation activities. 4
Educational support units schedule a day in the spring semester for unit leaders to discuss planning and evaluation activities with their respective units. Outcomes and Assessment All educational, administrative, and educational support units establish outcomes and conduct assessments to determine the extent to which those outcomes are being achieved. Each educational (workforce) program establishes program-level student learning outcomes, success standards, assessment activities, and timelines for measuring outcomes on the Unit Outcomes form. Conducted each fall semester, program reviews for workforce education units include revenue/expenditure analysis, number of majors, number of graduates, success (placement) rates, and licensure rates. Syllabi for workforce programs include student learning outcomes with accompanying activities and assessments for each learning outcome. Instructional units not classified as programs (transfer departments) identify student learning outcomes and accompanying activities and assessments within course syllabi. Administrative and educational support service units also identify expected outcomes, success standards, assessment activities, and timelines. For management of the AIP process, the RIE Committee has grouped units into the following divisions: Division Units 1-- Math/Health/ Agriculture, Biology, Chemistry/Geology, Engineering/Physics, Fitness Health Sciences Center, Kinesiology, Math, Nursing ADN, Physical Therapy Asst, Radiologic Tech, Surgical Tech 2 Liberal and Visual Arts, Dance & Music, Early Childhood Professions, Social & Fine Arts Behavioral Sciences, Communication and Theatre, Texas Shakespeare Festival, Photography 3 NA Original division 3 units were redistributed to other divisions. 4 Protective Continuing Ed-Kilgore, East Texas Police Academy, KC Fire Academy, Services Workforce Development 5 KC-Longview Dean, Adult Basic Education, Air Conditioning/Refrig, Continuing Ed- Longview, Cosmetology, Court Reporting, Diesel Tech, Emergency Medical Tech, LVN, Nurse Asst, Office Professionals, Small Business Development Center, Industrial Maintenance Tech 6 Business, Accounting, Auto Body, Auto Tech, Business Admin, Cisco, Computer Language, and Science, Corrosion Tech, Drafting, General Business, Language Technology Development: English, ESOL, Foreign Language, Developmental 7 Student Development 8 Administrative Services Engl/Read, Legal Asst, Management, OSHA, Process Tech, Welding Athletics, Counseling, Marketing, Print Shop, Financial Aid, International Students, Recruitment Services, Registrar, Special Populations/ADA, Student Life, Student Housing, Student Success, Testing, Trio, Job Placement Business Office, Instructional Leadership, Board of Trustees, Bookstore, Computer Center (IT), Institutional Advancement, Distance Learning, East Texas Oil Museum, Human Resources, Instructional Media, Institutional Research, Internal Audit (Board Liaison), Library, Physical Plant, Police Dept, President s Office, Risk Management 5
Annual Improvement Plan (AIP) Process When a unit identifies a gap between its desired outcomes and actual outcomes, it uses that information as the basis for an improvement plan which is reported on the Annual Improvement Plan (AIP) form. All educational, administrative, and educational support units, including those which engage in community service, participate in the AIP process. Early in the fall semester, units present their proposed AIP at a meeting which includes the unit head, the unit head s supervisor, the dean of institutional planning, the director of RIE, and any other interested personnel. These presentation meetings serve as a help session for those who need assistance in developing their AIP. They also inform the supervisor of the planning and evaluation activities conducted in the area. A third purpose of the presentation meetings is to facilitate quality control of the improvement process at the unit level. After improvement plans are discussed and adjustments are made to the plans, they are submitted to the supervisor and to the director of RIE. At the end of each unit s improvement plan cycle, which can occur at various points during the year, each unit submits an AIP Summary to the director of RIE and to the respective supervisor of the unit. During the fall semester, the dean of institutional planning compiles a summary report of all unit improvement plans implemented the previous academic year. KC has compiled an AIP Summary Report yearly since 1997; these reports are housed in the Office of Research and Institutional Effectiveness. The AIP Process is further demonstrated by the Annual Improvement Plan Flowchart, the Instructional Units Annual Improvement Plan Flowchart, and the Administrative & Educational Support Services Units AIP Flowchart. AIP Flowcharts and templates/directions for Unit Outcomes, Annual Improvement Plan, and Annual Improvement Plan Summary documents follow Kilgore College s Mission, Vision, and Core Values: Kilgore College s Mission and Vision Mission We prepare students for success in life by providing educational opportunities that have high quality, great value, easy access, and community focus. Vision We will lead through instruction, technology, and cultural development each characterized by creativity, innovation, and ingenuity. We will exhibit caring in all our relationships as we serve our students and our community. Given our redesigned mission statement, our core values are the following: high quality great value easy access community focus 6
Annual Improvement Plan Flowchart College Mission Core Values Administrative Unit Outcomes Educational Support Services Unit Outcomes Educational Units Student Learning Outcomes Engage in on-going assessment and evaluation of outcomes. Engage in on-going assessment and evaluation of student learning outcomes. Select one or more outcome needing improvement. Submit Annual Improvement Plan (AIP) indicating the issue, support data, improvement plan, target outcome, assessment method, budget implications, and timeline. Present AIP for feedback/approval. Conduct action plan and assessment; continue on-going assessment of outcomes. Submit AIP summary with assessment results and future action according to unit plan timeline. Note: All steps include consultation with supervisor, offices of institutional planning and research, and other support personnel as needed. 7
Instructional Units Annual Improvement Plan Flowchart Develop student learning outcomes (SLOs). [Workforce programs: Program-level SLOs and courselevel SLOs Transfer units: Course-level SLOs] Engage in on-going assessment and evaluation of learning outcomes. Select one or more outcome needing improvement. All steps include consultation with supervisor, office of institutional planning, office of institutional research, and other support personnel as needed. Submit Annual Improvement Plan (AIP) indicating issue, support data, improvement plan, assessment method, and timeline. Conduct action plan and assessment; continue ongoing assessment of student learning outcomes. According to unit timeline, submit AIP summary with assessment results and use of results for improvement. 8
Administrative & Educational Support Services Units AIP Flowchart Develop unit outcomes. Engage in on-going assessment and evaluation of unit outcomes. All steps include consultation with supervisor, office of institutional planning, office of institutional research, and other support personnel as needed. Select one or more outcome needing improvement. Submit Annual Improvement Plan (AIP) indicating issue, support data, improvement plan, target outcome, assessment method, budget implications, and timeline. Conduct action plan and assessment; continue ongoing assessment of unit outcomes. According to unit timeline, submit AIP summary with assessment results and use of results for improvement. 9
Unit Outcomes (Template) Unit Outcomes Unit Unit Purpose Statement Kilgore College Core Value(s) Outcome Success Standard Assessment Methods Assessment Schedule 10
Unit name of department, office, etc. Academic Year 2007-2008 Directions for Completing Unit Outcomes Unit Purpose Statement should concisely but comprehensively describe the essence of the unit, its primary purpose or mission. (The unit reviews this annually but generally does not change it unless there is a change in the scope of the unit.) Core Value one or more of the Kilgore College Core Values (high quality, great value, easy access, community focus) which the Unit Purpose Statement supports Outcome approximately 4 8 Workforce instructional units state outcomes in terms of student demonstration of learning (Students will...). Use verbs from Bloom s Taxonomy. These should be global learning outcomes that students should achieve by the end of their program of study. For this document, do not use enrollment, persistence, graduation, and licensure pass rates. You may use parts of licensure/certification tests if the scores can be broken down into specific learning outcomes. Administrative and educational support units state outcomes in terms of Efficiency turn-around time for requested materials ( Note: To use this type of outcome, units must have a way of documenting when requests are received.) Accuracy errors in an audit (internal or external) Number served students or other constituents Satisfaction general survey or user survey Awareness Do people even know your services exist? Student learning outcomes Consider who your learners are. Think about your daily activities. Look at your calendar. Who are your learners? What do you really want to have happen that would be an indicator of success for your unit? Success Standard usually a number or % Be realistic, but set a high standard (Emphasis Excellence). The goal is improvement, not rising to the height of a mediocre standard. Assessment Method(s) Make sure that assessments clearly yield data addressing the outcome. Use multiple assessments when possible. Administrative and Educational Support Units: Avoid relying on surveys alone. Assessment Schedule Determine the regular (systematic and ongoing) schedule which you will follow to gather these data: each semester, each year, or on a biennial basis. [Schedule for Analysis and Action Planning As a unit, when do you meet to determine what the actual results of your outcomes assessment mean (analysis)? Your unit s Annual Improvement Plan (AIP), due in September, addresses a deficiency you have found through the process of assessing your unit s desired outcomes.] Remember to make your planning and evaluation process meaningful and manageable. 11
Annual Improvement Plan (Template) Unit Academic Year Submitted by Unit Purpose Statement Institutional Core Value 1. Issue addressed in this plan 2. Data that identified this as an issue needing improvement 3. Improvement plan 4. Anticipated result of improvement plan 5. Method of measuring the improvement plan 6. Budget implications 7. Completion date 12
Directions for Completing the Annual Improvement Plan Unit name of department, office, etc. Academic Year 2007-2008 Unit Purpose Statement should concisely but comprehensively describe the essence of the unit, its primary purpose or mission. (The unit reviews this annually but generally does not change it unless there is a change in the scope of the unit.) Core Value one or more of the Kilgore College Core Values (high quality, great value, easy access, community focus) which the Unit Purpose Statement supports 1. Issue Instructional units will address student learning outcomes (SLOs). Administrative and educational support units will address other issues but should consider learning outcomes as well. Make sure that the issue you designate is one that addresses needed improvement in your area. Ex: Student demonstration of writing a thesis statement 2. Data You will have some information to support your selecting the outcome given in #1. This information could be from a survey, from a semester grade distribution, from a student focus group, etc. Use data from Institutional Research whenever possible. Ex: End of course evaluation indicated that 30% of students completing ENGL 1301 did not demonstrate correct use of a thesis statement in a multi-paragraph essay. 3. Improvement Plan Your improvement plan should be concise but should contain sufficient explanation to make it understandable to someone not familiar with your area. This is the action that you will take during the year to improve student learning (instructional areas) or processes (administrative and educational support areas). Ex: ENGL 1301 instructors will include an additional unit on writing thesis statements and will schedule individual consultations with students who do not demonstrate this outcome by mid-term. 4. Anticipated Result Define the target for your improvement plan. In most cases, this will be a number expressed in some format. Ex: The number of students demonstrating the correct use of a thesis statement will increase from 70% to 75%. 5. Method of Measurement Indicate what you will use to measure your improvement plan. For most units, this will be the same instrument etc. that identified the issue as needing improvement (see #2 above). Ex: ENGL 1301 end of course evaluation 6. Budget Implications What adjustments are needed within your unit s budget to implement this plan? (and/or) What new allocations are being requested? (Put None if the improvement plan does not require any reallocation of funds.) 7. Completion Date When will your unit complete this plan? Generally, this is January for fall improvement plans, May for improvement plans completed in the spring semester, and August for improvement plans completed in the summer. Note: Units should submit their AIP Summary to their supervisor and to the RIE Office at the completion of plan. 13
Annual Improvement Plan Summary (Template) Unit Academic Year Submitted by 1. Improvement plan synopsis a. issue b. action taken 2. Assessment method 3. Anticipated results 4. Actual results 5. Use of results for improvement 6. Future action required 14
Directions for Completing the Annual Improvement Plan Summary Unit name of department, office, etc. Academic Year 2007-2008 Unit Purpose Statement should concisely but comprehensively describe the essence of the unit, its primary purpose or mission. (The unit reviews this annually but generally does not change it unless there is a change in the scope of the unit.) Core Value one or more of the Kilgore College Core Values (high quality, great value, easy access, community focus) which the Unit Purpose Statement supports 1. AIP Synopsis Provide information regarding a. the issue (outcome) addressed Ex: Student demonstration of writing a thesis statement b. the action taken for improvement Ex: ENGL 1301 instructors will include an additional unit on writing thesis statements and will schedule individual consultations with students who do not demonstrate this outcome by mid-term. 2. Assessment Method examinations, grade distributions, portfolio evaluations, surveys, etc. Ex: ENGL 1301 end of course evaluation 3. Anticipated Results the target for this plan Ex: 75% of students completing ENGL 1301 will demonstrate the correct use of a thesis statement. 4. Actual Results usually expressed as a number or %; the actual results that were achieved when the assessment was conducted. Was the target outcome achieved? Was improvement indicated? Ex: 82% demonstrated the correct use of a thesis statement. The target was met. 5. Use of results for improvement (target outcome achieved) What is being changed or added to your procedures on an ongoing basis as a result of this improvement plan? Ex: ENGL 1301 instructors will continue scheduling individual conferences with students who do not demonstrate mastery of writing thesis statements by mid-term. (Enter See #6 if the target outcome was not achieved.) 6. Future action required (target outcome not achieved or further improvement desired) If the plan was not successful in accomplishing the desired outcome or if further improvement is desired, what future action is needed? Generally, this will serve as the impetus for the next AIP. (Enter See #5 if the target outcome was achieved.) 15