Organizing and Managing Program Assessment

Transcription

1 Organizing and Managing Program Dr. Robert L. Armacost Associate Professor Emeritus Dr. Julia J. Pet-Armacost Associate Dean Emerita University of Central Florida SACSCOC 2014 Institute on Quality Enhancement and Accreditation July 22, 2014 Robert L. Armacost and Julia Pet-Armacost, 2014 At the End of this Session You Will Be Able To: Describe program assessment concepts Describe key components of an effective program assessment management process Clearly articulate organizational components and support needed to successfully implement program assessment What are the pieces? What are the practices that are needed? July 22, 2014 Organizing and Managing Program 2 What Does Mean at Your Institution? A. Something required by SACSCOC B. What faculty do in the classroom C. Used for continuous improvement D. Used to evaluate performance E. Talk about it but really don t do it F. We do it but are not sure about the quality 0% 0% 0% 0% 0% 0% A. B. C. D. E. F. July 22, 2014 Organizing and Managing Program SACSCOC Institute 1

2 What Is Your Role in? A. Faculty conducting assessment in courses B. coordinator for program or unit C. IE/ office staff providing assessment support D. IE/ director managing assessment program E. Other 0% 0% 0% 0% 0% A. B. C. D. E. July 22, 2014 Organizing and Managing Program 4 Systems Supporting Improvement Institutional Effectiveness Program Strategic Planning Unit and Program Reviews Budget Linkages Share data and information Inform budget process Differences Different cycles Additional data elements Different purposes Continuous improvement Evaluation Planning July 22, 2014 Organizing and Managing Program 5 Definition Need a common understanding of assessment the examination and review of evidence representing levels of performance, accomplishment, and knowledge Purposes what you do with it Formative assessment focused on providing feedback to support improvement (e.g., not graded) Summative assessment focused on evaluating or judging the level of observed performance (e.g., graded) Important to separate formative from summative assessment reward PROCESS, not results July 22, 2014 Organizing and Managing Program SACSCOC Institute 2

3 Levels of Classroom assessment Determination of individual student performance at course level by instructors Course assessment Determination of how well a course is meeting student learning outcomes Program assessment Determination of how well an academic program is meeting student learning outcomes Determination of how well an educational support program is meeting its objectives Institutional assessment Determination of institutional performance July 22, 2014 Organizing and Managing Program 7 SACSCOC Requirements Comprehensive Standard The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness) CS educational programs, to include student learning outcomes CS administrative support services CS academic and student support services CS research within its mission, if appropriate CS community/public service within its mission, if appropriate July 22, 2014 Organizing and Managing Program 8 Program Citations 2013 Preliminary Results (n=75) Off-site On-site CS (educational programs) 64% 36% CS (administrative support) 52% 24% CS (academic and student support) 53% 29% CS (research) 21% 9% CS (community/public service) 52% 23% July 22, 2014 Organizing and Managing Program SACSCOC Institute 3

4 Institutional Questions Is there a need for some consistency among different program assessment plans? Example: college-wide competencies How does the institution know that programs are using assessment plans to improve programs? How does an institution improve the quality of program assessment plans? How does an institution provide evidence of compliance with assessment accreditation standards? July 22, 2014 Organizing and Managing Program 10 Effective Program Should Answer these Questions What and how does a program contribute to the development and growth of its students and/or the support of its customers? What outcomes are you trying to achieve? What is the level of performance? How, using the answers to the first three questions, can you improve the program? What actions did you take? What are the results of these actions? July 22, 2014 Organizing and Managing Program 11 Program Management Requires institutional management process Define scope academic and operational programs Develop assessment plan structure Establish completion and review schedule Provide organization and support July 22, 2014 Organizing and Managing Program SACSCOC Institute 4

5 Educational Outcomes Program System Institutional Context Coursework & Curricular Patterns Student Pre-college Traits Out-of-class Experience Educational Outcomes Classroom Experience *adapted from presentation by Dr. Gloria Rogers Timeline xx years Feedback Loop July 22, 2014 Organizing and Managing Program 13 Think System Exogenous Variables Controllable Inputs Environment Student Pre-college Traits Program System Institutional Context Coursework Program & Curricular Patterns Out-of-class Experience Classroom System Experience Timeline xx years Feedback Loop Educational Outcomes Outputs and Outcomes S U C C E S S Feedback Control System: group of interacting objects and processes working together to achieve a set of common purposes July 22, 2014 Organizing and Managing Program 14 Evidence of Program SUCCESS Sincerity means people believe the process has benefits Usefulness means the process improves programs Clarity means people understand the process Commitment means that leaders are supporters and people will do it Enthusiasm means the people want to do it Systemic and Sustainable means everyone is continuing to use it throughout the institution Support means people are not on their own help is available July 22, 2014 Organizing and Managing Program SACSCOC Institute 5

6 Which Element of SUCCESS Is Most Important to You? A. Sincerity B. Usefulness C. Clarity D. Commitment E. Enthusiasm F. Systemic and Sustainable G. Support 0% 0% 0% 0% 0% 0% 0% A. B. C. D. E. F. G. July 22, 2014 Organizing and Managing Program 16 Implementation Design Questions Who is required to conduct assessment? Academic departments, academic programs, divisions, offices What do you assess? Student outcomes, academic processes, student services When do you conduct assessment? Triennially, annually, each semester, monthly Where? Main campus, regional campuses, on-line Why? Scope of use How? Portfolios, surveys, institutional data, standardized tests July 22, 2014 Organizing and Managing Program 17 Essential Program System Elements Management objectives for assessment system Process design: content, focus, and mechanics of the program assessment process Quality assurance (QA) process Schedules and timelines for program assessment Documentation and support July 22, 2014 Organizing and Managing Program SACSCOC Institute 6

7 Management Objectives for the Program System Determine purpose of program assessment Accountability or program improvement or both Determine management focus outcomes or process or both Determine management involvement level Participation or oversight or both Determine management structures Centralized or decentralized or both At UCF July 22, 2014 Organizing and Managing Program 19 Essential Program System Elements Management objectives for assessment system Process design: content, focus, and mechanics of the program assessment process Quality assurance (QA) process Schedules and timelines for program assessment Documentation and support July 22, 2014 Organizing and Managing Program 20 Cycle Results Phase Assess Impact Take Action: Implement Change Mission/Purpose Educational Goals Identify Outcomes and Objectives PROGRAM Interpret Evidence Identify Measures and Methods Gather Evidence Planning Phase July 22, 2014 Organizing and Managing Program SACSCOC Institute 7

8 Mechanics of To improve, you need to know your purpose, where you are today and where you would like to go this is your PLAN Mission (purpose) Objectives or outcomes (what you need to achieve in order to get there) Measures (how well you are currently doing) To improve, you need to take action--results Analyze program or operations to determine changes Plan changes Take action Assess Impact July 22, 2014 Organizing and Managing Program 22 Student Learning is Complex Faculty Lab Assistants Spiritual Leaders Support Staff Library Staff Tutors Student Learning learning Resident Assistants Peers Teaching Assistants Mentors Advisers Coaches July 22, 2014 Organizing and Managing Program 23 Identify Scope of What should assessment include? Assess outcomes, performance or results; plan changes Assess resources (facilities, students, faculty) Assess processes (pedagogy, advising, feedback) Who receives feedback and how Academic program assessment typically focuses on student outcomes (learning and program) Administrative and support unit assessment typically focuses on quality of products, processes, services, and client/customer satisfaction with those services May also focus on student learning outcomes July 22, 2014 Organizing and Managing Program SACSCOC Institute 8

9 What Is Most Important Reason For Using a Single Model for? A. Eliminates ambiguity B. Makes planning easier C. Makes training easier D. Standardizes documentation E. Easier for evaluators to examine documentation 0% 0% 0% 0% 0% A. B. C. D. E. July 22, 2014 Organizing and Managing Program 25 Select a Model Popular model Nichol s 5-step approach A Practitioner s Handbook for Institutional Effectiveness and Student Outcomes Implementation, James O. Nichols, Agathon Press, 1995 Implement a consistent process across institution Helps increase comfort level Level of flexibility depends on maturity of process July 22, 2014 Organizing and Managing Program 26 Elements of an Plan Mission or purpose of program State primary functions and activities State why these are done (program purpose) Identify stakeholders Sufficient number of outcomes Program outcomes (student learning, operational) Three or more outcomes Multiple measures for each outcome Direct measures Indirect measures Two or more measures per outcome Methods for measurement and targets July 22, 2014 Organizing and Managing Program SACSCOC Institute 9

10 Should an Academic Department Have an Plan? A. Yes B. No C. Maybe 0% 0% 0% A. B. C. July 22, 2014 Organizing and Managing Program 28 How Often Should Plans Be Submitted and Reviewed? A. Once a year B. Every two years C. Every three years D. It depends 0% 0% 0% 0% A. B. C. D. July 22, 2014 Organizing and Managing Program 29 Essential Program System Elements Management objectives for assessment system Process design: content, focus, and mechanics of the program assessment process Quality assurance (QA) process Schedules and timelines for program assessment Documentation and support July 22, 2014 Organizing and Managing Program SACSCOC Institute 10

11 Quality Assurance Means A. All programs are doing assessment B. All programs are doing assessment well C. All assessment plans and results are reviewed D. Programs that are achieving outcomes are rewarded 0% 0% 0% 0% A. B. C. D. July 22, 2014 Organizing and Managing Program 31 Quality Assurance of the Process Quality assurance (QA) is needed to ensure that units and programs are following the process and doing it well QA function may provide one or more of the following Leadership of the assessment effort Management of the assessment process Monitoring function Feedback loop to improve the process Training Support Consultations July 22, 2014 Organizing and Managing Program 32 Implementing QA Setting goals for QA: cannot ask for complete, correct, and meaningful plans all at once Stage 1 Did everyone turn in documentation with all of the required pieces? Stage 2 Are the pieces there and are they done correctly? Stage 3 Are the pieces there, done correctly, and are they meaningful? You need to determine what you will monitor Examine the final product only Examine each piece of the process Examine the product at the end of each major step of the process (1. assessment plan development and 2. analysis of results and changes) Preferred July 22, 2014 Organizing and Managing Program SACSCOC Institute 11

12 Implementing a QA Process Determine how you will monitor Inspection or sampling Choice of method depends on resources and QA process chosen (checklist or other) What you monitor (every piece, each major step) Number of programs and units in the institution Type of feedback (evaluation or assessment) Determine how to obtain the documentation and schedule for reviews Critical success factor: make documentation preparation easy and facilitate the submission and QA review process July 22, 2014 Organizing and Managing Program 34 Who Should Conduct QA on Plans? A. Office B. University Committee C. Director of D. Peer Coordinators E. Other 0% 0% 0% 0% 0% A. B. C. D. E. July 22, 2014 Organizing and Managing Program 35 Essential Program System Elements Management objectives for assessment system Process design: content, focus, and mechanics of the program assessment process Quality assurance (QA) process Schedules and timelines for program assessment Documentation and support July 22, 2014 Organizing and Managing Program SACSCOC Institute 12

13 How Many Plans/Years Does a Program Work On At Once? A. One B. Two C. Three D. More than three 0% 0% 0% 0% A. B. C. D. July 22, 2014 Organizing and Managing Program 37 Scheduling Program Frequency How often will assessments be conducted? be reviewed? period--over what period will the measurements be taken? Flexibility for academic vs. administrative units Annual assessment cycle (recommended) Develop assessment plan for next year Measure and analyze the results for the current year Act on the results from a prior year and implement changes Submission and review schedule Annual Plan for next year and results from past year submitted together Semi-annual Two separate submissions Review of results informs plan preparation for next cycle July 22, 2014 Organizing and Managing Program 38 Serial Approach If you want a program or process to be measured for a full year, it takes three years to complete one cycle Plan for the assessment year Measure and analyze data at the end of the assessment year (results) and identify actions Act (implement changes) in the year following the assessment year Results would be reported every three years Year July 22, 2014 Organizing and Managing Program SACSCOC Institute 13

14 QA Implementation on a Serial Process Review assessment plan Review assessment plan Instruments Instruments Year Results and Use Results and Use Year July 22, 2014 Organizing and Managing Program 40 Schedule for Conducting Annual In any given year Develop assessment plan for next year Measure and analyze the results for the current year Document the impact of the changes made the prior year(s) Act on the results from a prior year and implement changes Three activities in parallel (Plan, Measure, Act) It is important to close the loop Use the results of assessment to implement changes Then measure impact of changes to affect your next plan July 22, 2014 Organizing and Managing Program 41 Parallel Approach If you want measures completed every year Year Year Year Communicating this concept is a major challenge July 22, 2014 Organizing and Managing Program SACSCOC Institute 14

15 QA Implementation on a Parallel Process (Dual Submission) Results Year Plan Measure Act Year Plan Year Measure Act Plan July 22, 2014 Organizing and Managing Program 43 QA Implementation on a Parallel Process (Single Submission) Plans, Results and Actions Year Plan Measure Act Year Plan Year Measure Act Report on results from previous year, planned use of results in current year, actual use of results in previous year, and assessment plan for current year July 22, 2014 Organizing and Managing Program 44 Essential Program System Elements Management objectives for assessment system Process design: content, focus, and mechanics of the program assessment process Quality assurance (QA) process Schedules and timelines for program assessment Documentation and support July 22, 2014 Organizing and Managing Program SACSCOC Institute 15

16 Documentation Required for reaffirmation of accreditation Needed for the QA function to do its job: improve the assessment process Challenges: Making it meaningful Balance between too much and too little Making it easy Making it accessible QA function Site visits Making it flexible to accommodate innovative approaches July 22, 2014 Organizing and Managing Program 46 Considerations When Developing a Documentation System process What elements need to be documented? Quality assurance Will you have a QA process and what will you document? Timelines When do individuals do the submissions and reviews? What type of history needs to be maintained? Access Who gets access to what elements? How do you maintain security? Technical approach Word/Excel templates Web-based templates Database July 22, 2014 Organizing and Managing Program 47 UCF s Website handbooks and organization Visitor access to all assessment plans and use of results July 22, 2014 Organizing and Managing Program SACSCOC Institute 16

17 TaskStream July 22, 2014 Organizing and Managing Program 49 WeaveEngaged July 22, 2014 Organizing and Managing Program 50 Other Web-Based Systems TK20 Nuventive TracDat LiveText July 22, 2014 Organizing and Managing Program SACSCOC Institute 17

18 Documentation What kind of documentation system do you use? What type of information is included in your documentation? plans results Feedback (QA reviews) Who has access to your documentation? What kinds of management reports are generated? Does the documentation system help to improve your assessment process? July 22, 2014 Organizing and Managing Program 52 Organization and Support Elements Decision authority Authorizes the process Approves major changes Quality Assurance authority Monitors process Recommends process, cycle, and schedule Doers Conduct assessment Document Support For doers For quality assurance function Support Support Decision Authority QA Authority Doers July 22, 2014 Organizing and Managing Program 53 Organizational Considerations Organization must be consistent with QA review level selected (frequency and type of review) Consider mix of academic and administrative personnel in committees and subcommittees Subcommittee members should have close ties with departmental and unit doers Subcommittees should have primary role in QA reviews Ideally headed by someone with faculty experience Credibility is an issue when working with academic programs Be sensitive to organizational differences Be sensitive to managerial and technical differences July 22, 2014 Organizing and Managing Program SACSCOC Institute 18

19 Director of Programs Director of Committee Programs Alternative Organizational Structures Director of Committee QA Review Committees Programs Committee Programs Committee QA Review Committees Programs Director of Director of July 22, 2014 Organizing and Managing Program 55 UCF Program System President University Committee Divisional Review Committee in each College and Administrative Division Coordinators for each program Department faculty and unit personnel Assess program learning outcomes July 22, 2014 Organizing and Managing Program 56 Committee Responsibilities University Committee Ensures that process is working correctly Coordinates quality assurance (QA) and serves a management review function Provides overall review of Divisional Review Committee recommendations Divisional Review Committee Responsible for QA at the division/college level Conducts specified QA reviews of own division/college programs Reports QA recommendations to UAC College Review Committee Academic Programs Chairs and Faculty QA reviews conducted here University Assesment Commmittee Review of QA process conducted here Divisional Review Commmittee Administrative Units Unit Heads and Staff July 22, 2014 Organizing and Managing Program SACSCOC Institute 19

20 Support for Support infrastructure Provides institutional and program surveys Provides technology and assessment expertise Supports programs and administration Training Creates common goals, understanding of assessment, consistency Builds commitment to assessment process Recognition Celebrates assessment success Consultations Can help you design a program assessment system May be limited to their assessment model July 22, 2014 Organizing and Managing Program 58 Organization of Support Function Level of resources Location of resources Person Office Dispersed throughout the organization responsibility Assigned or assumed Identify areas of existing or potential support Establish partnerships Establish communications and information exchanges Find ways to reduce workload Templates Web-enabled systems Survey support July 22, 2014 Organizing and Managing Program 59 Critical Success Factors Organization and Support Identify appropriate QA structure to facilitate selected assessment model or process Be sensitive regarding faculty and administrator roles Create active involvement at the doer level and encourage upward input Link support requirements to assessment process and QA regime Identify relevant support operations that can assist the assessment process win-win Develop support partnerships July 22, 2014 Organizing and Managing Program SACSCOC Institute 20

21 Summary of Program System Characteristics Create easily understood program assessment system Clearly defined content, schedule, and review process Develop manageable pieces plan and assessment results Minimize administrative workload for participants Easy on-line submissions and reviews Institutional surveys supporting assessment plans Training and assistance Produce useful results Improve student learning outcomes Improve operations Close the loop and share best practices July 22, 2014 Organizing and Managing Program 61 Self-: Defining Your Road Ahead Areas of Improvement Do you need to A. Clarify purpose of assessment? B. Clarify how to do assessment? C. Clarify who does assessment?? D. Create schedule for assessment? E. Create a quality assurance process? F. Provide for training, support, and documentation? G. Create reward system? July 22, 2014 Organizing and Managing Program 62 Which Area Requires the Most Work at Your Institution? A. Clarify purpose B. Clarify process C. Clarify who does it D. Create schedule E. Quality assurance F. Training, support, & documentation G. Rewards 0% 0% 0% 0% 0% 0% 0% A. B. C. D. E. F. G. July 22, 2014 Organizing and Managing Program SACSCOC Institute 21

22 Session Please complete the assessment form for the session. THANK YOU FOR YOUR ATTENDANCE AND PARTICIPATION July 22, 2014 Organizing and Managing Program 64 Questions Dr. Robert L. Armacost??? Dr. Julia Pet-Armacost Associate Professor Emeritus Associate Dean Emerita University of Central Florida University of Central Florida Higher Education Higher Education and Planning and Planning P.O. Box 2237 P.O. Box 2237 Flagler Beach, FL Flagler Beach, FL July 22, 2014 Organizing and Managing Program SACSCOC Institute 22