Great Expectations: Implementing a High-Stakes Testing Program in Maine. Background. Why High-Stakes Testing?

Transcription

1 Great Expectations: Implementing a High-Stakes Testing Program in Maine Kimberly Pukstas, PhD Leslie Rozeff, MSSW University of Southern Maine Background Maine hires roughly new caseworkers per year New hires complete a centralized training in Augusta Maine OCFS contracts with USM to provide training (currently two trainers) Training was five weeks long Training material was delivered face-toface in the classroom Why High-Stakes Testing? Two Goals: Ensure that training participants graduate with sufficient, entry-level mastery of child welfare fundamentals Increase credibility of the profession and training program 1

2 Evolution of a High-Stakes Process: Fall 2006 to Present Moving from knowledge test to holistic assessment of skills Creation of a statewide advisory committee Getting buy-in from management in Central Office and the Field Evolution of a High-Stakes Process: Fall 2006 to Present Structure job shadow experience Extend pre-service from 5 weeks to 8 weeks Supervisor involvement in job shadow oversight Development of tools & policies Development of online modules Policy Development Retesting Policies Option to test out Testing accommodations Dissemination of Results Confidentiality Feedback Loops Procedures for Updating Curriculum/Test 2

3 Summary of Testing Program Comprehensive approach assessing: 1) child welfare knowledge 2) requisite skill sets 3) adequacy of field experience Multiple raters Multiple formats, equal weighting Component 1: Knowledge Test Multiple Choice Exam (80 items) Online administration Helps determine if trainees are: 1) Informed on child welfare policies and practices 2) Educated on key issues 3) Completing online material Milestones Achieved: Buy-in from stakeholders Test bank developed Content expert reviews Online platform beta-testing Pen and paper pilots Trainee debriefing and comments 3

4 Challenges Ahead: Adaptation of items from other states Incorporating ongoing curriculum changes Obtaining adequate sample size Maintaining test security Pilot of online platform Component 2: Skills Assessments Standardized Evaluation Forms Administered by Pre-service Trainers Helps determine if trainees have the requisite skills in the three priority areas identified by OCFS: 1) Developing Documentation 2) Performing Client Interviews 3) Facilitation of Family Team Meetings Milestones Achieved: Buy-in from stakeholders Three assessment forms created Content expert reviews Standardization of skills exercises Increased training opportunities Increased feedback for trainees and supervisors 4

5 Challenges Ahead: Further standardization of exercises Assessing team meeting facilitation Inter-rater reliability Refinement of rating scales Training on the assessment tools Logistics of audio-visual recordings Component 3: Field Practice Workbook Weekly hands-on activities and observations in a variety of field settings Overseen and rated by Supervisors Helps determine if trainees have: 1) Sufficient field practice experience 2) Completed required activities 3) Routinely met with supervisor & colleagues Milestones Achieved Buy-in from OCFS Manual and support materials drafted Statewide training Piloted in two rounds Trainee debriefing and comments Supervisor debriefing and comments 5

6 Challenges Ahead Buy-in from supervisors Differentiating high-stakes materials from educational experiences Refinement of exercises Inter-rater reliability Refinement of rating scales Online platform Discussion Questions 1) What is the best method for developing stable test questions with a training curriculum in flux? 2) How do we balance the educational needs of the trainers with the training needs of the evaluators? 3) How to create realistic expectations from policymakers about high-stakes testing? 6