SOE Assessment Plan 9/17/12 SCHOOL OF EDUCATION COMPREHENSIVE ASSESSMENT PLAN

Transcription

1 SCHOOL OF EDUCATION COMPREHENSIVE ASSESSMENT PLAN Fall

2 Contents Introduction... 3 Mission, Vision, and Conceptual Framework... 3 Vision... 3 Mission... 3 Conceptual Framework... 3 Understanding Assessment... 4 Why Assessment?... 4 What Is Assessment?... 5 The Unit Comprehensive Assessment Plan... 6 Using the Conceptual Framework to Frame Assessment... 8 The Assessment Plan... 9 The Assessment Cycle Annual Assessment Activities Assessment Tools Program Improvement Plans Annual Assessment Retreat Program Improvement Implementation Guidance and Monitoring of the Assessment Process Program Assessment Plans Mission, Goals, and Objectives Standards Assessments Annual Process Accrediting Bodies Appendices

3 Introduction This document, in alignment with the conceptual framework, presents our Johns Hopkins School of Education Assessment Plan for all programs. This plan was developed through faculty and staff cooperation, collaboration, reflective practice, and by reviewing performance based assessment measures that ultimately drive program improvement and increase positive graduate outcomes. Vision, Mission, and Conceptual Framework Vision Founded in 1909 as the College Courses for Teachers, the Johns Hopkins University School of Education will lead the world in attracting the most talented and diverse individuals into the fields of education, counseling, and public safety. We will guarantee educational improvement and community well-being by assuring that our students, and others in the profession, have the most innovative tools and effective approaches to advance learning. Mission The Johns Hopkins School of Education's (SOE) mission is to support and advance the quality of education and human services for the continuous development of children, youth and adults. This addresses the Johns Hopkins University Mission to educate its students and cultivate their capacity for life-long learning, to foster independent and original research, and to bring the benefits of discovery to the world. The SOE attracts the most innovative and progressive scholars without disciplinary boundaries to solve complex educational problems through discovery and dissemination of new knowledge. Grounded in the Johns Hopkins tradition of research and innovation, its researchers and faculty explore the intersection of scientific research and how it can impact PK to higher education. In addition, school partnerships serve as living laboratories to test new and effective approaches to practice and leadership. Many of these partnerships have been recognized and adopted as national models. Conceptual Framework We accomplish this mission through: Research Teaching Program Development Leadership and Service Partnership Programs Our candidates are: Knowledgeable in their respective content area/discipline Reflective practitioners Committed to diversity 3

4 Data-based decision-makers Integrators of applied technology Understanding Assessment Why Assessment? Assessment is a culture of continuous improvement that parallels the School s focus on scholarship and research. Assessment ensures that students learn Assessment results provide evidence of the achievement of learning goals of each individual student when measured against established acceptable and target benchmarks. Each student s performance is measured through the assessment and benchmarked against either a standardized or assessmentspecific scoring guide. These assessments are used to monitor student learning achievement and to provide indicators to guide support and learning experiences. Assessment ensures that courses and programs are effective The analyses of assessments results provide indicators of courses effectiveness and guidance for courses development. The courses assessments are designed to provide evidence of student learning in relation to the objectives and outcomes associated with each specific course. Similarly, evidence across all the courses in a program provides indicators of the strengths and needs of programs to guide the effectiveness and relevance of courses. On a holistic level, these indicators of program effectiveness with data collected providing indicators of student learning are evidence of unit effectiveness. Another important function for measures of learning is to assist the school administration and individual faculty members to improve programs and courses. By measuring learning the school faculty can evaluate its students success at achieving learning goals, can use the measures to plan improvement efforts, and (depending on the type of measures) can provide feedback and guidance for individual students and individual programs. Assessment ensures that professional standards are met Assessment results provide evidence that professional standards such as, NCATE, and discipline specific standards are effectively addressed and that students demonstrate achievement of the standards relative to their learning goals. Assessment to demonstrate accountability (such as in accreditation) is an important reason to assess learning accomplishments. Measures of learning can assure external constituents such as potential students, trustees, public officials, supporters, and accreditors, that the organization meets its goals. Assessment ensures that the school s mission is addressed 4

5 Assessment results analyzed across the school provide indicators to ensure the effectiveness of learning activities in supporting the achievement of the school s mission. Program assessments aligned to the school mission provide evidence of the individual programs and students achieving the mission as outlined. Assessment evaluates how well a school accomplishes the educational aims at the core of its activities. The learning process is separate from the demonstration that students achieve learning goals. Do students achieve learning appropriate to the programs in which they participate? Do they have the knowledge and skills appropriate to their earned degrees? Few characteristics of the school are as important to stakeholders as knowing the accomplishment levels of the school's students when compared against the school's learning goals. What Is Assessment? Assessment is deciding what knowledge, skills, and attitudes/dispositions students need to learn and then ensuring that they learn them and are effective in applying them. Assessment is a process of continuous improvement that involves the following steps: Assessment Steps Set learning goals, objectives, and outcomes Design assessments to measure student learning achievement Provide authentic learning opportunities Assess student learning Collect and analyze results Use findings to inform decisions Implement evidence based changes Assess again Repeat continuously as a systemic process This process closes the loop by measuring the results of the changes being driven by the data collected. Assessment Levels Student Level The goal is to ensure that individual students achieve the designated outcomes for a course. Students must clearly understand the expectations and the benchmarks for their learning Varied opportunities for learning and assessment exist throughout the course Achievement is summarized in grades This is an individual focus Course Level The intent is to ensure that the students as a whole reach designated course outcomes Varied assignments and assessments from across the entire course Assignments and assessments are aggregated across students and course sections 5

6 Focus is on both students individually and the course as a whole Program Level Ensure that students as a whole achieve program level outcomes Key assessments are captured at key points in the program Assessments consist of summative internships, portfolios, capstones or comprehensive exams Focus is on both the students and the programs Unit Level Ensure that students as a whole achieve unit level outcomes Identified key assessments are captured at key points across all programs Assessments consist of common measures such as entry exams and post-graduation surveys Focus is on students and unit levels The Unit Comprehensive Assessment Plan Our assessment plan is aligned with our conceptual framework and includes the state and national standards that are appropriate to each of our programs. In keeping with our mission to support the continuous development of children, youth, and adults, our assessment plan provides information that is used to continuously and consistently assess our candidates progress. In addition, we believe that our assessment plan leads us to optimal programming and new models for educator preparation. The assessment plan was developed for the entire Unit or School of Education and consists of multiple measures that occur at transition points in the life of academic programs: admission, mid-program/preinternship, program completion, post-graduation (2 years), and post-graduation (5 years). The specific timing and nature of the evaluations differ slightly by program. All program and student assessments are stored in the data tool, TK20, for program faculty review. The specifics for each program are detailed in the Program Assessment Plan. Assessments of the teacher candidate through the transition points provide evidence of the candidate s growth and development. Each transition point uses various formative and summative assessments that collect sufficient data to determine the professional and pedagogical content knowledge of candidates and graduates. Formative evaluation (by course instructors, clinical supervisors, and advisors) gives the candidates ongoing feedback on their performance to identify aspects of their performance that need improvement with corrective suggestions. The summative assessments identify patterns and trends in candidate and unit performance and allows for programs and the unit to judge these summary statements against criteria to obtain performance ratings. Aggregate unit data and disaggregate program specific data are analyzed and distributed on an annual basis to members of the Assessment and Program Improvement Committee, faculty, field 6

7 coordinators, and the Dean. The faculty view this assessment process as an opportunity to revise specific content curricula and make program changes that result in furthering the development and growth of candidates, as they become Reflective Practitioners. Table 1 Major Assessment Points/Benchmarks ADMISSIONS Entry GPA GRE/SAT scores Admission demographics Personal essay Teaching experience Interview ratings Disposition Survey MID- PROGRAM/PRE- INTERNSHIP Course assignments Course grades Content verification E-Portfolio evaluation Academic plan Survey on diversity /inclusion dispositions Reflection on personal growth and goals Advisor/instructor input Student experience survey PROGRAM COMPLETION (CLINICAL EXPERIENCES) Course Grades Test results (such as PRAXIS II, CPCE Exam) E-Portfolio evaluation or final comprehensive exam or graduate project Survey on diversity/inclusion dispositions University Supervisor and Cooperating Teacher evaluations Course and Field Experience Assessment results POST- GRADUATION 2 YEARS OUT Employer survey Alumni survey School partner feedback MSDE data linked to our graduates POST- GRADUATION 5 YEARS OUT Employer survey School partner feedback MSDE data linked to our graduates Alumni survey Collect data from graduates through surveys and/or focus groups Exit interview or End of Program Evaluation 7

8 Table 2 Alignment of Conceptual Framework to Assessment Plan s Benchmarks Conceptual Framework Themes/Student Outcomes Key Assessment Points Data Points Knowledgeable in their respective content area/discipline Admission Mid-program Program Completion Post-Graduation Comprehensive Exams Praxis Exams Graduate Projects Reflective practitioners Mid Program Program Completion Post Graduation Committed to diversity Admissions Mid-Program Program Completion Post Graduation Data-based decision-makers Mid-Program Program Completion Post Graduation Integrators of applied technology Mid Program Program Completion Post Graduation Using the Conceptual Framework to Frame Assessment The School of Education vision guides the current conceptual framework. This includes the mission, initiatives, and student outcomes listed in the Introduction. SOE Mission: The Mission of the School of Education is to support and advance the quality of education and human services for the continuous development of children, youth, and adults. Initiatives: We accomplish this mission through: teaching, research, program development, leadership and service, and partnership programs. 8

9 Student Outcomes: Our candidates are: expert in their respective content area/discipline, reflective practitioners, committed to diversity, data-based decision-makers, and effective integrators of applied technology. We achieve our mission through performance of our program completers who demonstrate that they are: Content experts: understanding of content and demonstration of an ability to integrate appropriate national, state, and local standards in the candidate s work. Reflective in their practice: ability to modify, differentiate, and analyze effectiveness of instruction within the context of a particular classroom or school; ability to engage in and benefit from research that leads to improved student outcomes; cultivation of a reflective disposition for lifelong learning. Committed to diversity: respect for the differences among learners (e.g., but not limited to cultural, ethnic, racial, gender, sexual orientation, ability, socio-economic) in the candidate s classroom/school; commitment to the development of all students; commitment to working in schools characterized by diversity of students; understanding of different learning needs/styles and an ability to differentiate instruction appropriately. Data-based decision makers: understanding of the formal and informal assessments by which progress can be measured at the individual student, classroom, and school levels; ability to use data to plan for instruction (classroom based or school-wide) or program development. Effective integrators of applied technology: ability to enhance instruction through the appropriate integration of technology; ability to use technologies appropriate to school-based instruction. To address its mission the SOE has created a learner centered environment. Teaching and learning are focused on the achievement of the learner. The alignment of the vision, mission, initiatives, and outcomes is key to accomplish this. In the learner-centered environment, teaching and assessment are not separate but are intertwined as complementary processes to ensure learning. Key assessments, both stand-alone and embedded, are required at designated points during a student s tenure in a program. The analysis of student learning achievement performance, the effectiveness of the courses and the programs, and of the SOE unit is used to make changes to the assessments, courses (educational experiences), programs, and the unit using a process of continuous reflection. The Assessment Plan Built on the vision and mission of the SOE, the assessment plan is developmental and iterative. It is a system to ensure candidate learning achievement and continuous unit and program improvement. The conceptual framework guides the assessment process (Table 3). The assessment plan is. The resources and the impact described in the vision statement frame the assessment activities. Through effective teaching and innovative tools the SOE engages in excellent professional preparation (teaching and program development), high quality research, and innovative outreach (leadership and service, and partnership programs) initiatives. 9

10 In addition, each program is aligned with national, state, and professional standards. These standards are addressed through objectives and outcomes for each program, individual course, and educational experience. The aim is to demonstrate that SOe graduates are content knowledge experts, reflective practitioners, committed to diversity, data-based decision makers, and effective integrators of technology as evidenced in their knowledge, dispositions, and practice the mission-based student outcomes. To ensure this, student learning achievement is assessed at key points (admissions, midpoint/internship, graduation and post-graduation (through alumni and employer surveys, stakeholder feedback, and prek-12 student performance. The ultimate goal is to provide evidence of education improvement and community well-being as a result of student learning and impact, of course and program impact, and of unit impact. 10

11 Table 3: CONCEPTUAL FRAMEWORK LOGIC MODEL (DRAFT Spring 2012) Inputs Resource Capabilities Effective Teaching Innovative Tools Initiatives Excellent Professional Preparation High Quality Research Innovative Outreach The Johns Hopkins University Mission Statement CONCEPTUAL FRAMEWORK THE SCHOOL OF EDUCATION VISION SOE Mission Assessment of Learning Outcomes Key Points for Data Collection (For Assessments see Table 3) Domains Midpoint Program Post Grad Admit or Completion 2yr 5yr Internship Knowledge Disposition Practice Student Outcomes Content Experts Reflective practitioners Committed to Diversity Data Based Decision Makers Integrators of Applied Technology Outputs Impact Education Improvement Community Well-Being 11

12 Types of Assessments Critical assessments include measures that are direct and indirect, course embedded and stand-alone, and formative and summative. Direct assessments measure actual performance such as an individual student s score on a specific assessment, a score on a standardized test such as the Praxis I test, and university supervisor intern observations. Indirect measures include course evaluations, surveys, meeting notes, and stakeholders feedback. Course-embedded measures are assessments within required courses that expose students to systematic learning experiences designed to produce graduates with the particular knowledge or abilities designed to address identified student outcomes. These include test questions or assignments that are often discipline specific. Stand-alone testing or performance measures require students to demonstrate certain knowledge or skills as a requirement for graduation or at some other specific point in their degree program. These include such assessments such as portfolios, comprehensive exams, or standardized tests. Formative assessments provide feedback and guidance to candidates as they progress in their development toward achievement of mission-based student outcomes. Summative assessments capture performance at the five key points and are measured against expected levels of progress at each of those points. Analysis of both assessments informs changes in courses, programs, and unit operations. Key Assessment Points Admissions: Students are selected into a specific program based on a set of criteria that are (1) common across the school and (2) specific to that program. This selection is determined based on the knowledge or skills expected in graduates of their specified program. This includes content knowledge, skills, and dispositions. Midpoint: Students are assessed prior to entering the internship to determine the level of their performance in relation to that required for success in the internship placement. Program Completion: Students are assessed at the end of their program to determine their level of performance in relation to both the program and the unit goals and objectives. Post-graduation: Graduates complete surveys to indicate the impact their preparation has had on their performance. Employers and stakeholders provide information on graduates impact. Direct evidence of graduates impact on prek-12 student performance will be included when available. Permissions at Admissions At admissions, students are informed of the assessment process and its intent to ensure their learning and to continuously improve their programs. They are asked to give their consent to be included in this effort which means that (1) they understand that their assessments results will be studied to improve 12

13 their program and the SOE; (2) they give their permission to be surveyed as a student, at graduation, two years after graduation, and five years after graduation; and (3) they give permission to contact their employers both two and five years after they graduate. Contact information to accomplish this is obtained during their tenure as students and through the graduation application. 13

14 The Assessment Cycle Annual Assessment Activities A. SOE Program Reports Annual reports for each program summarize data from each of the five assessment benchmarks. These reports capture program improvement, data discussions among faculty (e.g., faculty and program meetings), the dissemination of data to stakeholders (e.g., students, community partners), and any other information used for program improvement. Lead faculty, field coordinators, Assistant Dean of Assessment, and the Associate Dean of Educator Preparation Programs coordinate these reports. Final reports are shared with the Dean s Office, SOE faculty, and the Assessment Committee. B. SOE Test Reports Annually, the Associate Dean of Administrative Services produces administrative reports. These reports include test summary results (e.g., PRAXIS, CPCE), admissions data, grade reports, and interview ratings. These reports are shared and disseminated to the Dean s Office, Assessment Committee, and SOE faculty. C. Field Experience Reports Produced annually are a number of reports facilitated by the Field Experience Office. This includes the entry and exit survey reports for programs, evaluation reports on clinical practices and internship evaluations. These reports are shared by the Field Experience Office and disseminated to the Dean s Office, Assessment Committee, and SOE faculty. D. SOE Alumni Survey Report Two-years and five-years after they complete their programs SOE alum are asked to share the impact of the experience in the unit and their programs on their careers through a survey that is administered each spring. The results are compiled and the data included in the deliberations of the assessment retreat. E. SOE Alumni Employer Report Two-years and five-years after they complete their programs SOE alumni are asked to provide contact information for their employers (commitments to this and permissions are secured during the admission process). A survey is sent each spring to identify employers to gather evidence of impact of program completers. The results are compiled and the data included in the deliberations of the assessment retreat. F. Program Accreditation Reports Any available program accreditation self-studies, accreditors reports, and annual updates for specific disciplines are analyzed and implications for unit and program improvement are entered into the deliberations at the annual assessment retreat. Presently the Counseling Program has CACREP Accreditation. 14

15 G. MSDE Program Completer Reports Reports compiled by MSDE on educator preparation metrics are analyzed and reviewed for unit and program improvement at the assessment annual retreat. Assessment Tools Data Collected Course data Field Experience data Meeting agendas and minutes Standardized and comprehensive test results Portfolios Alumni Survey Employers Survey (TBD) Stakeholders Survey (TBD) PreK-12 student data (when available) Data collection Tools Tk20 Tk20 provides a comprehensive outcomes assessment system that collects data systematically, compares data against specified outcomes/objectives, and generates detailed reports for compliance, analysis, and program improvement. Tk20 supports the managing outcomes-based assessment and measurement of student learning as well as institutional activities such as program improvement, institutional effectiveness. Assessment Analysis Tool The Assessment Analysis Tool is a customized program that assists faculty to convert exported Tk20 data into tables and graphs based on selected criteria. Assessment Components Each assessment has three components 1. Assessment description: This is the explanation of the assessment, the objectives that it addresses, and the directions for completing it. 2. Scoring guide (rubric): This is the list of criteria for the successful completion of the assessment. The scoring guide criteria provide guidelines for performance that is unacceptable, acceptable, or target. 3. Benchmark (standardized, embedded): This is the level of performance that is the goal for achievement. 15

16 Program Improvement Plans The assessment process occurs in an annual cycle that includes the collection of data at each of the key assessment point components. During the fall and spring semesters each year assessment data for each program is summarized in the Program Improvement Plan (PIP) that is submitted by June 1 st each year. This plan documents information related to candidate performance, program quality and unit operations. It is based on evidence from Tk20; program development, monitoring, and evaluation meetings; advisory groups, school/community partnerships; other stakeholder meetings; student feedback, IDEA evaluations, program metrics such as enrollments; program trends; and other data that provide evidence of program effectiveness and opportunities for program improvement. These plans are developed by a designated program faculty member(meeting agendas and minutes in relation to student performance, courses, and the program) in cooperation with the field coordinators (field experience summary data) and program faculty (course summary data) with the guidance of the Assistant Dean of Assessment and the Associate Dean of Educator Preparation Programs. Completed plans are shared with the SOE faculty, the API committee, the NCATE Task Force, the Vice Dean of Academic Affairs, and the Dean. Annual Assessment Retreat Each June an Assessment Retreat provides a review of the Program Improvement Plans to identify (1) levels of student learning achievement and (2) opportunities for improvement at both the unit and program levels. The retreat includes the program contacts for all the programs, program faculty, field coordinators, API Committee members, and the NCATE Task Force members. At the retreat the individual plans are analyzed to determine strengths and areas for improvement on candidate performance, program quality and unit operations. The outcome of the retreat is a set of recommendations to the Dean s Office based on the data reviewed. Program Improvement Implementation The Dean s Office reviews the Program Improvement Plans, the recommendations from the Assessment Retreat to determine changes to be implemented, resources to be allocated or reallocated, and target outcomes to be achieved are set. These changes are prepared or adjustments are made during July and August in preparation for implementation the next academic year. Target outcomes are established in relation to these changes and data is collected during the fall and spring semesters to determine the impact of the changes or additional changes to be made. Guidance and Monitoring of the Assessment Process Assessment and Program Improvement Committee The Academic Assessment Committee serves as the primary advisory body regarding the evaluation of student learning outcomes in graduate programs. The Committee also coordinates, but does not implement, the overall comprehensive assessment plan for the School. The Committee is responsible for providing feedback and support to faculty and academic leadership by reviewing assessment reports highlighting student learning outcomes and for identifying and disseminating promising assessment practices. The Assistant Dean of Assessment chairs this committee. NCATE Task Force 16

17 The Task Force oversees and is responsible for all NCATE related activities. The Task Force members provide accreditation information feedback to programs as well as bring issues from programs to the Committee. The Vice Dean of Academic Affairs and the Assistant Dean of Assessment co-chair this committee. THE FOLLOWING SECTION WILL BE DEVELOPED FOR EACH PROGRAM BY THE FACULTY ASSOCIATED WITH THAT PROGRAM Program Assessment Plans Mission, Goals, and Objectives Specific to the program Aligned with School vision and conceptual framework mission, initiatives, and student outcomes Standards NCATE SPA Standards CACREP InTASC Professional Development School Guidelines Assessments Alignment of assessments to all appropriate standards and to the conceptual framework Assessment descriptions with scoring guides and benchmarked outcomes Engagement of the faculty in their development Identification of opportunities for assessments that cross the programs Annual Process Review of assessment data and generation of recommendations Review of the assessments for needed change based on evidence Documentation of stakeholder engagement Faculty Students Partner Schools and organizations 17

18 Accrediting Bodies Middle States Commission on Higher Education The Middle States Commission on Higher Education is the unit of the Middle States Association of Colleges and Schools that accredits degree-granting colleges and universities in the Middle States region, which includes Delaware, the District of Columbia, Maryland, New Jersey, New York, Pennsylvania, Puerto Rico, the U.S. Virgin Islands, and several locations internationally. The Commission is a voluntary, non-governmental, membership association that defines, maintains, and promotes educational excellence across institutions with diverse missions, student populations, and resources. It examines each institution as a whole, rather than specific programs within institutions. National Council for Accreditation of Teacher Education- NCATE (soon to be CAEP) The National Council for Accreditation of Teacher Education (NCATE) is the profession s mechanism to help establish high quality teacher preparation. Through the process of professional accreditation of schools, colleges and departments of education, NCATE works to make a difference in the quality of teaching and teacher preparation today, tomorrow, and for the next century. NCATE s performancebased system of accreditation fosters competent classroom teachers and other educators who work to improve the education of all P-12 students. NCATE believes every student deserves a caring, competent, and highly qualified teacher. Specialized Professional Associations (SPA) ACEI CEC ELCC IRA ISTE NCSS NCTE NCTM NSTA Maryland State Department of Education (MSDE) Program Approval and Assessment 18

19 Program Approval and Assessment is one of three branches in the Division of Certification and Accreditation. Its purpose is to enhance teacher quality and provide assurance that Maryland's PreK-12 students are being taught by qualified, competent teachers. This branch implements The Redesign of Teacher Education and Title II federal reporting through program approval. Through sponsorship of the statewide professional development school network, the Branch promotes integrated school and teacher education reform. The Program Approval and Assessment Branch works closely with the Certification Branch, which awards state certification/licensure to qualified applicants. Council for Accreditation of Counseling and Related Educational Programs (CACREP) As a specialized accrediting body recognized by the Council for Higher Education Accreditation (CHEA), CACREP grants accredited status to graduate-level programs in the professional counseling field. The CACREP scope of accreditation includes the following program areas: Addiction Counseling Career Counseling Clinical Mental Health Counseling Marriage, Couple, and Family Counseling School Counseling Student Affairs and College Counseling Counselor Education and Supervision Appendices Assessment Cycle School of Education Conceptual Framework School of Education Program Improvement Plan 19

20 20

21 21

22 Johns Hopkins University School of Education Program Improvement Plan Program Name: Date of Program Meeting to Discuss/Approve Report: Data Sets Included in Review: Fall, Spring, Summer,, Fall, Spring Program Discussions of Assessment Data Reveal: Note: Indicate which data from Tk20 and other sources were analyzed in determining strengths and areas for improvement on Candidate Performance, Program Quality and Unit Operations. 1) Strengths of the Program: 2) Areas for Improvement in the Program: 22

23 3) Action Plan/Steps for Implementing Improvements: 4) Notes/Comments 23