A Guide to Teaching Strategies GOLD Interrater Reliability Certification

Transcription

1 A Guide to GOLD Interrater Reliability Certification Why Pursue Interrater Reliability Certification? GOLD is an authentic, ongoing, observation-based assessment that relies heavily on your judgment as a teacher. It is the information you collect each day from observing children as they work and play within the context of everyday meaningful experiences that makes this assessment so robust and effective. With this evidence in mind, you use development and learning progressions to evaluate each child s progress and compare it with widely held expectations for children s learning and development. The results of your evaluation inform the decisions you make when planning developmentally appropriate learning experiences for individual children and your group as a whole. It is important to be considered reliable (accurate) in your assessments in part because your program administrator will likely use your data in some way for reporting purposes. One criticism of any observation-based assessment system is confidence in the evaluator s judgments. Administrators of programs using this type of assessment have voiced concerns about whether evaluators (teachers) are marking observational assessments accurately. They wonder if two teachers observing the same situation would rate the child the same way, and more importantly, if they would mark it the same way as the developer would mark it. Perhaps the most important reason to be considered a reliable evaluator is to be able to make your assessment decisions with confidence confidence that you have the information you need to plan and respond appropriately for all the children in your classroom. To address administrators concerns and support your efforts, GOLD has taken steps to increase the accuracy of your assessment ratings through an interrater reliability certification. This process will measure the degree of agreement between the rater of the assessment (you) and the developers of the assessment. If you are using the assessment to inform instruction, then your ratings must be accurate in order for the intervention or teaching strategies you use to be effective.

2 We developed the Interrater Reliability Certification to increase the reliability of your judgments as you use GOLD improve planning for individual children by helping you assess children accurately identify professional development needs using data collected in the interrater reliability certification process The Interrater Reliability Certfication is neither designed nor intended to evaluate you as a teacher. Instead, it may be considered one of many competencies you could possess that will benefit the children you work so hard to support each day in your classroom. What Is Interrater Reliability? After completing the GOLD implementation training, you may choose to advance your implementation skills and receive an Interrater Reliability Certification from. The online certification process gives you the opportunity to evaluate multiple child portfolios and compare your ratings with those of master raters. To achieve interrater reliability, you must reach a score of 80% reliable in each area of development and learning. You will have the option of choosing interrater reliability for children in one of three age-groups: birth through age 2, age 3 through kindergarten, or birth through kindergarten. In statistics, interrater reliability is the extent to which two or more individuals (observers/raters) agree. It addresses the consistency of the implementation of a rating system. For this certification test, each child s portfolio has been reviewed and rated by master raters. They have agreed on a single rating for each objective or dimension. Reliability on a checkpoint rating means that your rating is within a range of ±1 of the master rating. For example, if you enter a rating of 5 on a dimension and the master rating is 6, the rating is within the range of reliability. If you answer 4 and the master rating is 6, the rating does not fall within the range of reliability. In order to be certified as reliable, at least 80% or more of the ratings in each area of development and learning must fall within the range of reliability (±1). A Guide to GOLD Interrater Reliability Certification Page 2

3 Interrater Reliability Certification Process You will begin by evaluating three portfolios. If upon completion of the first round of portfolios you do not reach 80% reliability in one or more areas of development and learning, you will be tested on those areas again in a second round by evaluating three additional portfolios. If upon completion of the second round of portfolios there are still areas in which you have not reached 80% reliability, you will evaluate a third set of portfolios in those areas, and so on, until you have reached 80% reliability in each area. There is no limit on the number of rounds. The portfolios include documentation and supporting evidence to sufficiently rate each child in the six areas of development and learning: Social Emotional, Physical, Cognitive, Language, Literacy, and Mathematics. Please note that none of the children are new to the program and none have had excessive absences; therefore, none of the dimensions should be rated Not Observed. Testing Agreement The GOLD Interrater Reliability Certification test is meant to be taken only by the individual named in the account. As an examinee, you should neither solicit nor utilize any outside assistance during the testing process. In order to take the test you will be required to agree to a statement certifying that 1) you are the individual who is registered to take the test, and 2) you are completing the test without assistance from any other source. Preparing for the Test To prepare for the interrater reliability certification, recommends that you complete the GOLD Implementation Training prior to beginning the certification process. provides two forms of implementation training: an online professional development course relating to GOLD, which is available to all users via the Professional Development tab in the Dashboard; or a 2-day inperson professional development session. Training sessions introduce participants to the structure of GOLD, the objectives for development and learning, the assessment cycle, and the evaluation of child portfolios. Additional preparation is recommended and available through the Practice Checkpoints that are part of the online professional development module. In the Practice Checkpoints users can review portfolios, enter checkpoint levels, and then compare their levels to those established by master raters. Users will also see a justification, which explains the reason behind the master rating. Finally, users can prepare for the certification test by reviewing Objectives for Development & Learning in order to become more familiar with the 38 objectives, the dimensions, and the progressions of development and learning. A Guide to GOLD Interrater Reliability Certification Page 3

4 Testing Timeline You may take as much time as you need to complete each round there are no time limits. Your start and completion dates for each round will be listed in your Dashboard under the Professional Development tab. We suggest that you take time between rounds to review the objectives for development and learning in the areas you did not pass. This also is a good time to review the professional development training modules again. How to Complete the Test To complete the Interrater Reliability Certification process, log in to your TeachingStrategies.com account and navigate to the Professional Development tab on your Dashboard page. There you will see the Interrater Reliability menu item on the left side of the screen. Click on it to access the Interrater Reliability screen. You will be asked to read and agree to the verification statement. If you are unable to agree to the statement for any reason, please consult with your administrator or send an to implementation@teachingstrategies.com for clarification. A Guide to GOLD Interrater Reliability Certification Page 4

5 Choose one of the following age-groups for testing: birth through age 2, age 3 through kindergarten, or mixed ages. If you are not sure which age-group to select, check with your administrator. The screen that follows is your launching point for the certification test. It includes a list of the rounds to be completed, your progress to date, and details of each round. The start date and completion date are listed as well as the areas you need to complete in order to pass the test. Click on Round 1 to begin. A Guide to GOLD Interrater Reliability Certification Page 5

6 This will take you to the Checkpoint by Area screen. The Checkpoint by Area screen is similar to the screens in GOLD. You will see three children listed down the left side of the chart and the areas of development and learning across the top. The boxes in the middle of the chart show your completion status in each area. Refer to the Key on the right side of the screen for details. You may review and change your ratings at any time before you submit them to be scored. Once you have completed all the checkpoint ratings for all children in an area and have reviewed your ratings, click Submit to see your results. To begin rating, click on any box in the chart. This will take you to the Checkpoint screen. Here you will see the progression (with indicators), examples, and color bands. Click on View All in the lower left corner of the Checkpoint screen to see the documentation on that child related to that progression. Review the documentation and choose a level. Proceed to the next dimension and continue until you have completed all the dimensions for that child in that area. Note that you may stop at any time by clicking Save and Close in the lower right corner of your screen. You may also leave the test, log out of your account, and return to the test at any time. A Guide to GOLD Interrater Reliability Certification Page 6

7 The View Results screen will show your results in each round and area. Once you have completed all the areas in a given round, you will see your interrater reliability results for each completed area. You will be given an overall reliability score for each area of development and learning. Your results will not include specific information about each rating as compared to the master rating. Additionally, you will be provided with suggestions for next steps before you begin your next round, if applicable. Once you have passed all the areas of development and learning, you will receive access to a link that allows you to print a Certificate of Completion (in PDF format) for your records. Your certification will be valid for three years from the date of completion. In most cases, the results of your certification test will be made available to your administrator through the GOLD system. Please note that if you paid an individual testing fee, your results will not be made available to your administrator. A Guide to GOLD Interrater Reliability Certification Page 7