Leveraging Technology to Gather, Analyze, and Use Assessment Data for Program Improvement WASC Conference April 12, 2013

Transcription

1 Leveraging Technology to Gather, Analyze, and Use Assessment Data for Program Improvement WASC Conference April 12, 2013 Dr. Kathy Theuer, Associate Dean and Director of Accreditation Dr. Carla Piper, Assessment Coordinator and Instructional Designer 1

2 Brandman University Part of the Chapman University System Private/Non-Profit School of Education, 1 of 5 Schools 26 campuses across California, Washington & Online Offer Fully Online & Blended/Hybrid Courses 2

3 Credential Programs Multiple Subject Teaching Credential Single Subject Teaching Credential Special Education Teaching Credential Administrative Services Credential Pupil Personnel Services Credentials School Counseling School Psychology 3

4 Degree Programs Bachelor of Arts in Early Childhood Education Master of Arts in Teaching (MAT) Master of Arts in Education (MAE) Master of Arts in Educational Leadership and Administration Master of Arts in Counseling Ed.S in School Psychology Master of Arts in Special Education Doctor of Education in Organizational Leadership 4

5 Accreditation 5

6 The Challenge Gather and store data from multiple sources Aggregate data by program Disaggregate data by: Campus Delivery model (blended/fully online) Term/Academic Year Analyze data and generate reports for multiple accrediting bodies 6

7 Creating an Assessment Infrastructure Platforms People Processes 7

8 The Platforms Assessment Management System Paired with our Learning Management System 8

9 Processes What data is collected? Who gathers the data? When is data collected? Where is data stored? Who analyzes data? Who provides technology training? Who addresses technology issues? Who monitors participation in the assessment process? 9

10 Assessment Team Director of Accreditation- Oversees assessment process, coordinates data collection, analysis and report writing, collaborates with assessment team members Assessment Coordinator- Builds and copies rubrics each term, generates reports, oversees calibration of instructors Accreditation Technical Coordinator- Manages fieldwork and TPA sites, addresses technical issues with LiveText 10

11 Assessment Team Accreditation Team- Reviews data with curriculum teams, shares data with stakeholders and prepares accreditation reports Center for Instructional Innovation- Provides training for students and faculty on the use of LiveText 11

12 Data Sources Course Embedded Signature Assignments linked to PLO s Student Disposition Assessments Fieldwork Evaluations/Field Placement Data Demonstration of Mastery Portfolios Teacher Performance Assessments 12

13 Course Embedded Assessments Example: Signature Assignment EDUU 609 Curriculum Comparative Analysis Paper Student View- Blackboard Student/Instructor View in LiveText Generating Reports Instructor Level Program Level 13

14 Student View in Blackboard 14

15 Student View in LiveText 15

16 Instructor View in LiveText 16

17 Instructor/Student View in LiveText 17

18 Instructor/Student View 18

19 Instructor Report 19

20 Aggregated Results 20

21 Disaggregated Results 21

22 Inter-Rater Reliability 22

23 Disposition Assessments Excerpt from Brandman s School of Education Professional Code of Conduct The National Council for the Accreditation of Teacher Education (NCATE) defines dispositions as professional attitudes, values and beliefs demonstrated through both verbal and nonverbal behaviors as educators interact with students, families, colleagues and communities. These positive behaviors support student learning and development. The School of Education at Brandman University has adopted a set of professional dispositions that we feel are essential for prospective educators to possess and demonstrate throughout the credential/master s program. The dispositions adopted by Brandman University s School of Education are listed below. The behaviors listed are examples of the kind of conduct expected of students in the School of Education, but are not intended to be inclusive. Professional Demeanor & Responsibility - punctual and prepared for classes and field experiences, dresses appropriately in professional settings, responsible for meeting deadlines and following instructions, accepts responsibility for own actions, capable of dealing with disappointment, flexible, adapts to change, reacts appropriately under stress, exhibits a positive attitude toward schools, teaching, students and parents Commitment to Learning for ALL Students - shows sensitivity to all students, respects individual differences, responds to the needs of students and their families and communities, embraces diversity to improve student learning, demonstrates compassion, patience and fairness Communication - has command of written language (mechanics, usage, spelling) and oral English language, uses standard English in writing and speaking, demonstrates responsive listening skills, able to communicate with a variety of audiences Collaboration - is skilled in expressing self in groups, works effectively with professional colleagues and other adults, able to accept and respect different points of view Self Reflection - engages in problem solving and self-evaluation, accepts and acts upon constructive criticism and is receptive to new ideas, uses feedback to make improvements, strives for personal and professional growth Ethics - demonstrates truthfulness and honesty, maintains ethical and legal behaviors in interactions with others, maintains confidentiality, respects the intellectual property of others by giving credit and avoiding plagiarism/cheating 23

24 Aggregated Results 24

25 Disaggregated Results 25

26 Fieldwork Placement Data - Create a database of partnerships statewide (over 500 signed partnerships) - Assign Master Teachers and University Supervisors - Provide a shared workspace for student teacher/master teacher/supervisor - Evaluate student performance 26

27 27

28 ) Fieldwork Shared Workspace 28

29 Fieldwork Assessment 29

30 Demonstration of Mastery Portfolios 1. Student completes online application form in LiveText 2. Student builds and submits portfolio in LiveText 3. Technical Coordinator matches student with with assessors 4. Portfolio is assessed by multiple reviewers 5. Technical Coordinator lets student know if they have passed 6. Student records are updated accordingly 30

31 Portfolio Application Form 31

32 Portfolio Template 32

33 Portfolio Template 33

34 Assigning Assessors 34

35 Scoring Portfolios 35

36 Teacher Performance Assessments 1. Student completes online application form in LiveText 2. Student submits TPA 3. Technical Coordinator matches student with assessors 4. Technical Coordinator lets student know if they have passed 5. Student records are updated accordingly 36

37 TPA Application Form 37

38 Teacher Performance Assessments 38

39 Assigning Assessors 39

40 TPA Rubric 40

41 Sharing Data with WASC Program Reviewers 41

42 Benefits of Technology LiveText has allowed us to: Gather and store assessment data easily Establish consistent expectations of instructors across multiple campuses Assess inter-rater reliability Monitor instructor and student participation in assessment activities 42

43 Benefits of Technology Develop a database of fieldwork sites and placements Utilize multiple assessors for demonstration of mastery portfolios and teacher performance assessments Conduct blind reviews for portfolios and teacher performance assessments 43

44 Challenges of Technology Training for students & faculty Getting students registered for LiveText Monitoring instructor use of LiveText Revising rubrics 44

45 Questions??? For more information contact: Kathy Theuer Carla Piper 45