Choosing a Progress Monitoring Tool That Works for You. Whitney Donaldson Kellie Kim Sarah Short

Transcription

1 Choosing a Progress Monitoring Tool That Works for You Whitney Donaldson Kellie Kim Sarah Short

2 Goals To describe who we are and what we do as a Center To describe the technical review process To review our website and learn how to maximize the use of the Tools Chart To review the decision making process

3 What is the National Center on Student Progress Monitoring? Funded by the U.S. Department of Education, Office of Special Education Programs National technical assistance and dissemination center Housed at the American Institutes for Research, in conjunction with researchers at Vanderbilt University

4 Our Mission To provide technical assistance to states and districts and disseminate information about progress monitoring practices proven to work in different academic content areas (Gr. K-5).

5 Goals Raise Knowledge and Awareness Provide Implementation Support Provide for National Dissemination

6 Raise Knowledge and Awareness Forming partnerships States, Districts, Associations, Technical assistance providers, Institutions of higher education, Other interested groups

7 Provide implementation support Technical Review Committee Regional Trainers Web Services

8 National Dissemination Developing resources Supporting on-going information sharing Summer Institute advanced web services regional meetings

9 Technical Review Committee and Review Process

10 Technical Review Committee Dr. Chris Espin, University of Minnesota Dr. John Hintze, University of Massachusetts Dr. Robert Linn, University of Colorado Dr. Chris Schatschneider, Florida State University Dr. Ed Shapiro, Lehigh University Dr. Mark Shinn, National-Louis University

11 Procedural Integrity Disclose all contractual obligations or affiliations Independent review by two reviewers for each tool Discrepancies in preliminary review were discussed by conference call to reach consensus Vendors were offered opportunities to respond to the initial review results Re-review of new/additional evidence

12 Seven Core Standards of Technical Adequacy Foundational Psychometric Standards 1. Reliability 2. Validity

13 Seven Core Standards of Technical Adequacy (Continued) Progress Monitoring Standards 3. Sufficient number of alternate forms 4. Sensitivity to learning 5. Evidence of instructional utility 6. Specification of adequate growth 7. Description of benchmarks for adequate end-of-year performance or goal setting process

14

15 Dynamic Indicators COST of Early Basic ACCOMMODATIONS TECHNOLOGY, RESOURCES, Literacy Skills: HUMAN AND FOR Nonsense Word Fluency, 6th Edition (DIBELS NWF) Cost SPECIAL NEEDS SERVICE & SUPPORT APPLICATION/IMPLEMENTATION Sopris per West materials per year: are ISSUES $2.36 and subsequent includes grade $1.40 level per materials student per DIBELS years. student for This for year for all price 1 Internet full Testers materials. of access is required for can training. use administer will of product Paraprofessionals require the test. services. 1-4 hours Group, $1.00 DIBELS maintain, benchmark, per scores. data summarize student progress This system per includes and to year for Accommodations: A list Dynamic Inc. Measurement The DIBELS Approved is available Palm Pilot version available standardized, the Administration and at: administered the monitoring reports. Sopris complete costs and are: kit unlimited Scoring Guide (e.g., giving principle correspondence DIBELS NWF is a directions primary language the student s or sign Roland Good to individually and years. printed $35.00 This benchmarks, for includes subsequent for $59.00 year progress pre-for language). Natalie Flint which test of alphabetic Phone: /2399 common including letter-sound grades blend letters into and words of in ability monitoring and 1 A materials a with student Braille Braille. has may edition sufficient Scores also of be stimulus for used skills the if Field are available tested training and should manuals The letters represent their most provide all needed randomly sounds for students in administration guide students. Forms materials and materials and for scoring Braille edition may not be implementation information. nonsense K-2. sheet downloaded and 25 directly comparable. A included assistance DIBELS web-based and website. is available technical tutorial is student is presented with letter ordered VC and CVC at the verbally words on an 8.5 x11 whole of paper and asked to example, the individual vaj sound of each letter /j/ produce, read, the a nonsense word. For correct. total say the of the three student if the word letter-sounds stimulus could vaj say, word to obtain /v/ is /a/ photocopied the DIBELS for website. free from can be The to as is produced the produce he/she student number can, as is many of and allowed letter-sounds the letter-sounds final one minute tool provides correctly information one minute. score student The nationally of 39,000 test performance was Kindergartners representative normed using English. sample and a on 37,000 first graders. Implementation Information DIBELS-NWF minutes EASE OF takes USE administer students. alternate per forms There probe individual available are to 262 progress Raw, developmental percentile monitoring. benchmark and for scores Raw adding sounds scores are available. one score minute. the produced number are A composite calculated correctly of letter- by instructional recommendations is provided along obtained with in by achieving outcomes research. calculating from subsequent the longitudinal odds literacy of The correct middle Benchmark on benchmark of letter first goals sounds grade. goal are by is based 50 subsequent odds of achieving early literacy the goals.

16 Decision Rule Each standard is comprised of two or three essential components Each and every component MUST demonstrate sufficient evidence to be rated as sufficient standard

17 (1) Frequent Progress Monitoring Alternate Forms are of equal and controlled difficulty. Number of alternate forms is sufficient for frequent (at least monthly) progress monitoring. TRC Decision Yes Yes Meets evidence standards Any other patterns of responses Does not meet evidence standards (2) Adequate growth or goal setting Adequate growth or goal setting is specified. Basis for measuring adequate growth or goal setting is specified. Basis can be norm-referenced, criterionreferenced, or other appropriate procedures Evidence is based on specifically using this tool OR is based on test construction principles from tools used in referred studies. TRC Decision Yes Yes Yes Meets evidence standards Any other patterns of responses Does not meet evidence standards

18 Benchmarks for Adequate End-of-Year Performance or Goal Setting Benchmarks for adequate end-of-year performance or goal Yes setting are specified. Basis for identifying benchmarks for adequate end-of-year performance or goal setting is specified. Basis can be normreferenced, criterion-referenced, or other appropriate Yes procedures. Evidence is based on specifically using this tool OR is based on test construction principles from tools used in referred studies. Meets evidence TRC Decision standards Improved Teacher Planning or Student Achievement Provided evidence shows improved teacher planning or Yes student achievement by using the specific tool. Evidence/data are research based. Yes Meets evidence TRC Decision standards Yes Any other patterns of responses Does not meet evidence standards Any other patterns of responses Does not meet evidence standards

19 Data Sensitivity Use of the specific tool produces data that are sensitive to children s development of academic competence and/or to the effects of intervention. Evidence/data are research based. TRC Decision Yes Yes Meets evidence standards Any other patterns of responses Does not meet evidence standards Reliability Reliability estimates (e.g., Cronbach s alpha, test-retest, and/or inter-rater reliability) are reported. Provided information is relevant and specific to the tool submitted. TRC Decision Validity Validity information (e.g., content, concurrent, predictive, and/or construct) is reported. Provided information is relevant and specific to the tool submitted. TRC Decision Yes Yes Meets evidence standards Yes Yes Meets evidence standards Any other patterns of responses Does not meet evidence standards Any other patterns of responses Does not meet evidence standards

20 Next Steps 2006 Review of SPM Tools in based on General Outcomes Measures, Mastery Measurement, or both is in progress

21 National Center on Student Progress Monitoring Website and Tools Chart

22

23

24

25

26

27

28 Which Tool Should I Choose? A Look at Possible Decision Making Scenarios

29 I m interested in monitoring student progress in mathematics in my district for grades 1-3. Which tools would be appropriate? AIMSWeb Monitoring Basic Skills Progress (MBSP) Yearly Progress Pro STAR

30 Which tools have probes that can be administered to students using paperand-pencil? AIMSWeb Dynamic Indicators of Basic Early Literacy Skills (DIBELS) EdCheckup Test of Word Reading Efficiency (TOWRE) Test of Silent Word Reading Fluency (TOSWRF) *Please note: In order to use the full features of each tool, such as graphing and generating reports, many tools require internet access.

31 Which tools offer accommodations for students with special needs ( e.g. English Language Learners, hearing impaired)? Dynamic Indicators of Early Basic Literary Skills (DIBELS) EdCheckup STAR

32 I am a teacher with 25 students in my classroom and cannot afford to set aside blocks of time to administer progress monitoring probes to selected students. Are there tools that can be administered by the paraprofessional in my classroom? AIMSWeb Dynamic Indicators of Early Basic Literary Skills (DIBELS) EdCheckup Monitoring Basic Skills Progress (MBSP) Yearly Progress Pro STAR Test of Word Reading Efficiency (TOWRE) Test of Silent Word Reading Fluency (TOSWRF)

33 I am interested in finding a tool that I can use to monitor my students progress weekly or even more frequently. Are there tools that have at least 20 alternate forms? AIMSWeb Dynamic Indicators of Early Basic Literary Skills (DIBELS) EdCheckup Monitoring Basic Skills Progress (MBSP) Yearly Progress Pro STAR