Discovery Education Assessment K-HS Benchmark Assessments in ELA and Mathematics

Transcription

1 Discovery Education Assessment K-HS Benchmark Assessments in ELA and Mathematics What validation data or reports are available? Are technical specifications available for the assessments? Please see the following Technical Manual for validation studies and psychometric information pertaining to Discovery Education s interim assessments. Additional reports and technical evidence is available upon request.

2 COMMON CORE INTERIM BENCHMARK TECHNICAL MANUAL APRIL 2013

3 Discovery Education Assessment Common Core Interim Benchmark Technical Manual Table of Contents I. Introduction: Discovery Education Assessment II. Assessment Standards and Content Validity A. Common Core Standards B. Test Development and Review C. Web Alignment III. Test Administration Online IV. Test and Item Scores and Test Reports A. Test and Item Scores B. Interim Assessment Reports C. Interim Assessment Reports Examples V. Reliability, Proficiency Levels, Validity and Growth A. Test Reliability B. Proficiency Levels C. Validity D. Vertical Scale Averages and Growth Appendices A. Test and Question Statistics, Reliability and Scale Scores B. Web Alignment Study Page 1

4 I. Introduction: Discovery Education Assessment Discovery Education has focused on the use of formative assessments to improve K-12 student learning and performance. Bridging the gap between university research and classroom practice, Discovery Education Assessment offers effective and user-friendly assessment products that provide classroom teachers and students with the feedback needed to strategically adapt their teaching and learning activities throughout the school year. Discovery Education Assessment has pioneered a unique approach to formative assessments using a scientifically research-based continuous improvement model that maps diagnostic assessments to each state s high stakes summative tests. Discovery Education Assessment s Predictive Test-Specific Interim Assessments are aligned to the content assessed by each summative assessment allowing teachers to track student progress toward the standards and objectives used for accountability purposes. Furthermore, Discovery Education Assessment subscribes to the Standards for Educational and Psychological Testing articulated by the consortium of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. This technical manual presents information about the Common Core Interim Assessments used during and school year. Discovery Education Assessment across the United States In , K-2 students in the United States took over 3.3 million Common Core interim benchmark assessments created by Discovery Education Assessment. These students were found in twenty one different states across the country. As more states transition to the Common Core State Standards, more students are being assessed with Discovery Education Assessment s Common Core Interim Benchmark Assessments. Page 2

5 II. Assessment Standards and Content Validity Content validity evidence shows that test content is appropriate for the particular constructs that are being measured. Content validity is measured by agreement among subject matter experts about test material and alignment to state standards, by highly reliable training procedures for item writers, by thorough reviews of test material for accuracy and lack of bias, and by examination of depth of knowledge of test questions. To ensure content validity of all tests, Discovery Education Assessment carefully aligns the content of its assessments to a given state s content standards and the content sampled by the respective high stakes test. Discovery Education Assessment hereby employs one of the leading alignment research methodologies, the Webb Alignment Tool (WAT), which has continually supported the alignment of our tests to state specific content standards both in breadth (i.e., amount of standards and objectives sampled) and depth (i.e., cognitive complexity of standards and objectives). All Discovery Education Assessment tests are thus state specific and feature matching reporting categories of a given state s large-scale assessment used for accountability purposes. Common Core Standards The Common Core State Standards Initiative is a state-led effort to establish a shared set of educational standards in English language arts and mathematics for grades K-12. The standards are adopted voluntarily by state, and are designed to prepare students to be ready to enter college or join the workforce upon graduation. These standards will not be tested until the school year. Since the implementation of the Common Core standards, Discovery Education Assessment is working with the adopted states to help transition to these new standards from their old state standards. Below are the Common Core State Standards that can be found on the interim benchmark assessments in Reading and Mathematics, grades 3-8, and Algebra 1 & 2 in High School. Common Core English Language Arts Reporting Categories Reading: Literature Reading: Informational Text Reading: Foundational Skills English Language Arts Standards : Writing English Language Arts Standards: Language English Language Arts Standards: Listening & Speaking Common Core Algebra 1 & 2 Reporting Categories Number & Quantity Algebra Functions Statistics & Probability Page 3

6 Common Core Mathematics Reporting Categories Operations & Algebraic Thinking Number & Operations in Base Ten Number & Operations Fractions Measurement & Data Ratios & Proportional Relationships The Number System Expressions & Equations Geometry Statistics & Probability Test Development and Review Alignment For the last nine years, Discovery Education has led in assuring educators that its items are aligned specifically to each state s curriculum or district pacing guide. Every change in a state s curriculum was carefully identified and items changed to meet the current year s statewide assessment. Alignment is done by trained test developers who have subject matter expertise, teaching experience in the grade/subject, and the assessment expertise to produce appropriate items. Each individual has certification in the grade/subject and has at least three years teaching experience in the area. The alignment is managed within our software and mapped one item at a time. This is not a software alignment but rather a teacher who compares each item to the state standard to determine if it is aligned. We begin by matching our tests to the Common Core State Standards that are assessed. We agree that for teachers to have confidence in the results, they must be assured that an assessment is aligned to their standards. The test has to exhibit content validity, which is demonstrated when test items represent the subject area, such as math or reading. In other words, a math assessment must have items that match or align to the Common Core Standards and benchmarks defined by a state s curriculum and high stakes test. Difficulty levels are based on actual prior student performance and provide teachers with a crucial comparison of how a current class or individual student compares to what is generally expected of student performance on these items. Discovery Education Assessment pioneered a unique approach to benchmark assessments using a scientifically research-based continuous improvement model that maps diagnostic assessments to each state s high stakes test. Discovery Education s Predictive Test-Specific Benchmark tests are aligned to the content assessed by each state test allowing teachers to track student progress toward the standards and objectives used for accountability purposes. This same predictive approach is being applied to state tests as they transition to Common Core assessments. Page 4

7 Items Discovery Education employs only certified, experienced teachers with content majors and master s degrees to align, create, and develop items; the content review, copy editing, and quality control departments are also staffed by competent, qualified teachers with graduate degrees. Discovery Education intentionally employs teachers with familiarity in varied subjects, age groups, and ability levels, which gives the item development teams an impressive range of expertise. They work with psychometric staff to review and systematically match items to Common Core standards. All items have appropriate psychometric properties from field testing that permit accurate, valid, and reliable predictive tests. Our commitment to rely on competent, seasoned educators throughout the entire development process assures that items and tests are accurate, appropriate, and accessible. Discovery Education content team begins the test development process with the state standards, test blueprints and test specifications. We begin by matching our items to the Common Core standards that are assessed. No software tool can sufficiently match items to state standards. Curriculum experts must do this job one item at a time. Discovery Education Assessment s software facilitates the curriculum expert s job of aligning each item to a Common Core standard. This task is re-aligned every year, using prior year student performance statistics to assure continuous alignment, reliability, and validity. All Common Core standards are loaded into the Discovery Education Assessment tool, which allows our curriculum experts to build state-specific tests. Item notes and field test data are available at the time of item selection for each grade and subject test. Bias Statistics All Discovery Education assessments incorporate a systematic, official statistical bias analysis, using Rasch analysis on gender, ethnicity, and differing abilities. Discovery Education Assessment is committed to assuring students, teachers, and administrators that we are sensitive to and cognizant of the need for assessments to be bias-free. While it is important to know that a test measures what it is purported to measure, it is just as important to know what a test does not measure. Discovery Education assessments are designed and reviewed to guard against culture or gender bias and to address issues of disability. Types of Items Discovery Education assessments feature multiple choice questions that measure the maximum range of cognitive skills in the content areas. Using multiple choice questions reduces the cost, test taking time, and provides immediate results with diagnostic and predictive capabilities. However, we generally include constructive response open response and performance tasks with the assessments that could be manually scored and used by teachers in the classrooms for formative purposes. In addition, Discovery Education services provide access to performance tasks and tools for locally created items. Page 5

8 Refreshing Item Pool Discovery Education Assessment routinely works with schools to field test new benchmark items to replenish our pool of available items. This approach assures that we continuously have field tested items available. This improvement process also assists our test coordinators to receive specific feedback on every item, thereby further increasing the reliability of each item included. Quality Control Quality control is a crucial aspect of Discovery Education Assessment s approach to item construction. The quality control process ensures that every component of a Discovery product is consistent and accurate within and across printed or online versions of tests. The quality control department determines that the online and printed versions of tests match exactly, that reports are accessible and complete, and that scores on Discovery Education tests are correlated to the proficiency or mastery specifications provided by the state. A key role of the quality control department involves testing the functionality of the online interface and examining reports for accuracy. Copy Editing & Proofreading A vital part of the development team is the finalization team, responsible for ensuring the typographical accuracy of all assessments. The copy editing personnel proofread the assessments after they are released from the content review team. The editors correct any typographical or mechanical errors that appear in the test, and they also look for errors in the layout and placement of graphics, instructions, page numbers, or margins. Copy editors also perform a final examination of formatting to ensure that each test is formatted to match the state s high stakes tests formatting. When the assessment is proofread and necessary changes are made, the test is converted into PDF format, then proofread again. If any errors still exist, they are corrected before a copy of the final test is posted for printing. The proof from the printer is then edited one final time. These multiple instances of thorough proofreading enable Discovery Education to produce tests that are not only exemplary in content but also accurate in grammar and mechanics. Overview of Benchmark Item Review Steps Items are reviewed multiple times by experienced teachers, a psychometrician, a grammar expert, and Director of Testing for spelling errors, errors in usage, and awkward phrasing. All items are reviewed by at least 2 reviewers to have a single correct answer and appropriate distracters. All items are reviewed for grade appropriateness in content and readability, using Flesch- Kincaid scale. Items have field test and actual administration data to support reliability of grade level in terms of difficulty and content validity. Item Depth of (DOK) is reviewed and displayed for educators on reports. Page 6

9 Where items are expected to include vocabulary above grade level, items are reviewed to assure that context clues are on grade level. In math items, required computations are reviewed to be appropriate for grade level and appropriate to the time-constraints of the formative environment. Items that require critical thinking skills are measured in terms of steps required and the difficulty levels to provide an appropriate mix across the skill area. Many items require graphics to support the thinking skill measurement. These items are reviewed to assure that the graphics print and appear on the web with clarity, appropriate level of detail, and appropriate grade level to measure the skill. Graphic items are kept simple enough to assure web display within 2-3 seconds but complex enough to measure the skill or accomplishment being tested. Item answers are varied to distribute them randomly across answer options A, B, C, and D so that no discernable pattern is possible for correct answers. Web Alignment Discovery Education contracted with an independent research team, Test Prep, led by Dr. Michael K. Smith in April 2012 to conduct a Web alignment study of the Discovery Education interim benchmark assessments with the Common Core State Standards. This process is repeated after any revisions are made to the interim benchmark assessments. The WAT (Web Alignment Tool) version 2 was used to measure categorical concurrence, depth-ofknowledge consistency, range-of-knowledge correspondence and balance of representation. A summary of the results are below. Results of this study are being used to make appropriate revisions to the interim benchmarks. A plan of action has already been set in place to increase depth-of-knowledge consistency in the reading assessments and the categorical concurrence in the mathematics assessments. For more details on this alignment study, please see Appendix B for the document Web Alignment Study of Discovery Education Assessment Benchmarks with Common Core Standards. Page 7

10 Categorical Mathematics Tests Alignment Summary YES WEAK NO TOTAL # % # % # % # 59 63% 34 37% % 0 0% 0 0% % 3 3% 0 0% % 2 2% 0 0% 93 Categorical Reading Tests Alignment Summary YES WEAK NO TOTAL # % # % # % # 58 82% 13 18% % 8 11% 6 8% % 9 13% 0 0% % 1 1% 0 0% 71 Page 8

11 III. Test Administration Online To administer interim benchmark tests online an administrator must first import students to the Discovery Education website ( Once students are imported and populated into grade pool and teacher classes, testing can begin at any time after the recommended window opens. For steps on importing students, populating classes, and administering interim benchmarks online, please refer to the Assessment User Guide. This document is located at: and on the help site at discoveryeducation.com. Page 9

12 IV. Test and Item Scores and Test Reports Test and Item Scores Discovery Education Assessment reports the following item and test scores on its Interim Assessments. Student Level Scores: Test Percent Correct: Test Number Correct: Reporting Category Percent Correct: Reporting Category Number Correct: Vertical Scale Score: State Percentile: National Percentile: Proficiency Prediction: The percent correct on an interim benchmark The number correct on an interim benchmark The percent correct on a particular reporting category The number correct on a particular reporting category A scale score on a 1000 to 2000 scale The percent of students that score lower than a particular scale score. The state percentile is based on all students in a particular state that completed an interim assessment The percent of students that score lower than a particular scale score. The national percentile is based on a stratified random national sampling of students who completed an interim assessment A prediction of the level of student performance Class, Grade, School, and District Level Scores: Item Percent Correct: Item Percent Incorrect: Test Mean Number Correct: Test Mean Percent Correct: Vertical Scale Mean: The percent of students in a class, grade, school or district that answer an item correctly The percent of students in a class, grade, school, or district that answer an item incorrectly The arithmetic mean (average) of the number correct on an interim assessment for a class, grade, school, or district The arithmetic mean (average) of the percent correct on an interim assessment for a class, grade, school, or district The arithmetic mean (average) of the vertical scale score on an interim assessment for a class, grade, school, or district Page 10

13 Proficiency Level Number: Proficiency Percent: Median State Percentile: The number of students in each proficiency level on an interim assessment for a class, grade, school, or district The percent of students in each proficiency level on an interim assessment for a class, grade, school, or district The middle value of all state percentiles for a school or district Interim Assessment Reports Discovery Education produces multiple standard report formats to coincide with each interim assessment, as well as dynamic district reports. Interactive reports are linked to recommended digital remediation selected from Discovery Education s award-winning streaming. Discovery Education recognizes the tremendous value in prompt, easy to read reports that allow all stakeholders, including students, parents, teachers, administrators and district staff, to instantly determine how learning is progressing. While the software and standard reports are not customizable, all reports allow easy export of data to CSV or EXCEL file formats for flexible reporting or import into other reporting tools. Discovery Education Assessment reports the following: For Teachers: For Administrators: Proficiency predictions by subject point to which students are at risk. Predictions of proficiency levels within each Common Core standard define specifically what to focus on in remediation. Performance Indicator results viewed with the test items to define detailed gaps in student thinking processes. Growth of student performance across time and comparable across grade, school, and district. Additionally, digital instructional resources, targeted by Performance Predictor, which teachers can assign. Summarized grade, school, and district learning status for each subject and standard by proficiency level. Growth of student performance across time and comparable across grade, school, and district. Identification of all student results by Common Core standards to examine possibilities for professional development and need for new texts and resources. Guided priority for instruction of demographic subgroups and examination of results of special programs. For community leaders, summarized status of each school for comparison purposes, with information during the year similar to state assessment reports. Page 11

14 For Students and Parents: Predictions of student proficiency by subject Growth of individual performance across time and compared to school and district performance Detail of skills and subskills mastered or in need of remediation Individual student responses by item Access to engaging digital content aligned to targeted skills and objectives Interim Assessment Example Reports This section provides annotated examples of each of the following Interim Assessment Reports: Class and Grade Skill Summary Report Student Skill Report Student Sub-Skill Report A and B Item Summary Report Answers Report Individual Student Report Drill Down Report Comparison Report School Comparison Report Scale Comparison Report Subgroup/Disaggregated Reports Comparative Growth Reports Page 12

15 School Reports Class and Grade Skill Summary Report The Class Summary Report identifies performance by skill for entire class or grade. Using the red, yellow, and green stop light approach (Common Core also has blue), proficiency is shown for each standard within reading, math, and science. In this example, the bar chart displays the percent of students Level 1 (red), Level 2 (yellow), Level 3 (green), and Level 4 (blue) by each of the reporting categories for Reading. The actual percentages for each proficiency level for each reporting category are given in the table below the bar chart. Student Skill Report The student skill report uses the same color-coded approach to plot individual student performance and proficiency by skill. In this example, the level of performance on each skill for each student is presented. Furthermore, in the two far left columns, the student overall proficiency level is presented together with the number correct on the interim assessment. Page 13

16 Student Sub-skill Reports This report displays performance on every Common Core sub-skill measured. Pale green means it is correct, and the letter gives the student s incorrect answer. This feature helps the teacher identify why the student selected the wrong answer. Little teacher time is required to go through the whole set of reports. Item Summary Report The item summary report presents information on every question in an interim assessment. The summary is tallied over a class or grade. For each question, the following information is provided: the correct answer; the number and percent correct; the number and percent incorrect; the Common Core reporting category (skill) and reporting subcategory (sub-skill); and the level of difficulty of the question (easy, medium, or hard). This report is also available interactively. This interactive feature allows a user to sort by any of the information above and to search for resources tied to the skill or sub-skill questioned such as streaming videos and quizzes. Page 14

17 Answer Report The answer report provides each student s specific response (ABCD) to each question on an interim assessment. In the top row, the correct answers on the assessment are provided. Then, each student s individual choices (ABCD) on each question are listed. Summary information is provided on the righthand side: number correct, percent correct, state percentile rank, and vertical scale score. The student s overall subject proficiency is indicated by the highlighted color. Page 15

18 Individual Student Report The individual student report summarizes scores on all interim assessments for an individual student. The Overall Subject Summary (box to the right) displays the Number Correct, Total Questions in Test, Percent Correct, Scale Score, and State Percentile for Test A, Test B, and Test C for this student. This student has started at Level 2 (yellow) on Test A and has stayed consistently at that level on Test B and C. A national percentile, based on Test B, is also included. The Growth chart on the left graphs the student s scores on the three assessments along with the school and district averages. Furthermore, the solid gray line represents an End-of-Year Target score; to reach Level 3 at the end of the school year, a student would need a scale score at or above this value. The Performance by Standard Summary table displays the Proficiency level of this student by each of the five Mathematics standards. Finally, the student s answers to all questions are provided in the last table. Page 16

19 Drill-Down Report District Administrators can compare schools. Both District and School Administrators can use a series of drill down reports by grade, teacher, or student, and sort by proficiency prediction. They can also get Microsoft Excel extractions of data, view comparisons across NCLB sub-group populations, and track progress of all classes and schools. Page 17

20 Comparison Report This report compares students across multiple testing periods and monitors student progress during the year where they have recently concentrated instruction and where students have not retained learning from earlier instruction. Page 18

21 Comparative Growth Report This report helps teachers compare students with each other at the district or school level. This report will be available when at least two benchmark tests have been completed. There are three sections to this report. Test 1 and Test 2 Regression Each student is represented by a sphere. A larger sphere indicates multiple students with the same score on both tests. In the teacher version, you can mouse over the sphere to display the student name. The horizontal axis is the first test and the vertical access is the second test. The vertical scores shown on each axis is defined by the range of each proficiency level on that particular test. As a comparison, the grid boxes with color show all of the students who scored within the same proficiency level on both tests. The heavy line across the chart shows the regression line based on the students who took these particular tests for either the school or the district. A regression line makes predictions of scores based on the scores on a prior test. The dotted lines represent the error bands of the regression line. This is the lower and upper bound values of the standard error of estimate. For example, look at the chart above. If a student scores a 1631 on test 1, we expect them to score somewhere approximately between 1611 and 1693 on test 2. A student scoring a 1499 on test 1, we only expect to score approximately between 1479 and 1561 on the second test. You can individually view the school line in red or the district line in blue on this section of the report. You also have the option of viewing them together at the same time; just select how you would like to view in the Select Report Criteria section. If students are outside of the dotted lines, they have scored higher or lower than the expected range on the second test based on how they scored on the first test. The students names shown as small red spheres scored below their expected vertical score range on the second test. The ones shown as small green spheres scored above their expected vertical score range on the second test. All of the gray spheres represent students with a vertical score in their expected range. If both lines are shown, students will be compared to the district line. Page 19

22 Test 1 to Test 2 Distribution of Change This bar chart allows you to see the distribution of scores from the regression line. The higher above or below the 0 line, the farther away that score is from the regression line. This is calculated by the actual score on test 2 minus the expected score on test 2. This chart also displays the standard error of the estimate (SEE). The standard error of estimate is a measure of the average distance from the regression line, or the accuracy of the predictions. Notice that those scores falling above or below the SEE lines are the ones displayed as red or green. These are the scores identified as above or below average. Ranked Order of Change All of the students names are listed and organized in three groups: Below Average Students, Above Average Students and Average Students. This section displays the scores on the first and second test, color coded to display which proficiency level their vertical score is associated with. Students appear in ranked order according to their residual scores, that is, the difference between their actual test 2 scores and their expected test 2 scores. Page 20

23 District Reports School Comparison Report This report summarizes district performance by proficiency level on a specific interim assessment together with performance on the reporting categories (skills) that comprise that assessment. This report is broken down by school to show comparisons across the district. This graph presents an example of a Grade 3 Reading interim assessment. Scale Comparison Report The scale comparison report is a table that displays the average scale scores for each of the tests and the average change for the whole district and each individual school. It is broken down by grades and subjects. Page 21

24 Detail Export The detail export is a.csv or.xls report with student level data. This report has all the different data points for each test the student has taken including overall subject and skill proficiency levels, number & percent correct, state & national percentiles and vertical scores. Page 22

25 V. Reliability, Proficiency Predictions, Validity and Growth Test Reliability Test reliability provides evidence that test questions are consistently measuring a given construct, such as mathematics ability or reading comprehension. Furthermore, high test reliability indicates that the measurement error for a test is low. Reliabilities are calculated using Cronbach s alpha. The following tables present test reliabilities and sample sizes for Discovery Education Assessments for three time periods Fall (Test A), Winter (Test B), and early Spring (Test C)) in the subject areas of Reading and Mathematics for The median reading reliability was.85, with median sample sizes of 52,628. The median mathematics reliabilities for the three time periods was.79 with a median sample size of 49, Common Core Sample & Reliability Coefficients: Reading Test A Test B Test C N Reliability N Reliability N Reliability Kindergarten 44, , , Grade 1 83, , , Grade 2 77, , , Grade 3 55, , , Grade 4 59, , , Grade 5 59, , , Grade 6 58, , , Grade 7 56, , , Grade 8 55, , , English 1 16, , English 2 11, , Page 23

26 Common Core Sample & Reliability Coefficients: Math Test A Test B Test C N Reliability N Reliability N Reliability Kindergarten 43, , , Grade 1 82, , , Grade 2 77, , , Grade 3 55, , , Grade 4 59, , , Grade 5 59, , , Grade 6 49, , , Grade 7 56, , , Grade 8 54, , , Algebra 1 11, , Algebra 2 12, , Geometry 10, , Page 24

27 % of Students Proficiency Predictions Discovery Education Assessment provides each student with a prediction of proficiency status on each of their benchmark assessments. These interim benchmarks are designed to measure skills tested by the future PARCC and SBAC summative assessments but in a more time-limited format. The results of these benchmarks are intended to provide teachers, administrators, and students with reliable and valid predictors of student performance. Discovery Education Assessment uses the Continue-to-Learn model for providing proficiency predictions. The Continue-to-Learn model differs from an Absolute model in that it does not give a prediction based on if the student took the state test at the same time. Instead, this gives a prediction based on that student continuing to learn throughout the rest of the year up to the time of their state test. With this model, there are fairly stable distributions of proficiency levels throughout the year. The Continue-to-Learn model helps teachers identify at the beginning of the year those students who need the most instruction and in what area that instruction is needed. With an Absolute model, a large percentage of students would score as Not Proficient in the beginning of the year because many of the skills have not yet been taught. Discovery assessments feature four performance levels. There are no state proficiency levels to predict so we must rely on Discovery created criterion referenced cut scores. A large representative sample of scores on the Common Core assessments will be separated into four performance groups or levels. Twenty percent of scores are identified for Level 1, thirty percent for Level 2, thirty percent for Level 3, and twenty percent for Level 4. Discovery Education Assessment: 20/30/30/20 Model Blue (Level 4) indicates a performance level achieved by the top twenty percent of students on the Common Core assessment. Students at this level may be ready to learn more advanced standards or to broaden their knowledge of the grade level standards. 30% 25% 20% 15% 10% 5% 0% Level 1 Level 2 Level 3 Level 4 Green (Level 3) indicates a good performance level, at or slightly above grade level. Yellow (Level 2) is tied to a performance level at or slightly below grade level. Red (Level 1) suggests a poor performance level. The lowest fifth of all test scores are at this level. The student with Red scores may need significant support to achieve the grade level standards specified by the Kentucky Department of Education. Some schools will consider these students for additional assessments and Tier 2 or Tier 3 instructional strategies under an RTI model. Page 25

28 Validity Area Under a ROC Curve Area under a ROC curve is a measure of the discrimination of a test, or the ability of an assessment to correctly classify those students as at-risk. AUC values above.90 are excellent, between are good, and between are fair. During the Fall of 2012, Discovery Education conducted a ROC analysis on the Common Core interim assessments from the school year. AUC values ranged from , with a median value of Area Under the Curve: Common Core Test A Test B Test C Area Std. Error Area Std. Error Area Std. Error Reading Math Predictive and Concurrent Validity Predictive and concurrent validity are criterion-related validity methods that measure the correlation between a test and pre-validated assessment. In the fall of 2012, Discovery Education partnered with a district from the Commonwealth of Kentucky, a state that implemented a fully aligned Common Core summative assessment in the spring of The school district of 12 schools provided their studentlevel summative assessment data to Discovery Education for analyses. Correlations between state summative scale scores and scale scores from the Discovery fall and winter (predictive) assessments and the spring (concurrent) assessments were calculated. Reading predictive validities ranged from while concurrent validities ranged from , all were significant (p <.01). Math predictive validities ranged from while concurrent validities ranged from , all were significant (p <.01). Page 26

29 Predictive and Concurrent Validity Statistics Reading Grade Predictive (A) Predictive (B) Concurrent (C ) 3 Correlation N Correlation N Correlation N Correlation N Correlation N Correlation N Math 3 Correlation N Correlation N Correlation N Correlation N Correlation N Correlation N All correlations significant at the 0.01 level (2-tailed). Proficiency Prediction Score The Proficiency Prediction Score is used to determine the accuracy of predicted proficiency status. Under the NCLB legislation, it is important that states and school districts help students progress from a Not Proficient status to one of Proficient. The Proficiency Prediction Score is based on the percentage of correct proficiency classifications (Not Proficient/Proficient). If a state uses two or more classifications for Proficient (such as Proficient and Advanced ), the percentage of students in these two or more categories would be added together. Also, if a state uses two or more categories for Not Proficient (such as Below Basic and Basic ), the percentage of students in these two or more categories would be added together. To see how to use this score, let s assume a school district had the following data based on its annual state test and a Discovery Education Assessment Spring benchmark assessment. Let s use data from a Grade 4 Mathematics Test as an example: Page 27

30 Predicted Percent Proficient or higher = 70% Actual Percent Proficient or higher on the State Test = 80% The error rate for these predictions is as follows: Error Rate = /Actual Percent Proficient - Predicted Percent Proficient/ Error Rate = 80% - 70% = 10% In this example, Discovery Education Assessment under predicted the percent of students proficient by 10%. The absolute value (the symbols / / ) of the error rate is used to account for cases where Discovery Education Assessment over predicts the percent of students proficient and the calculation is negative (e.g., Actual - Predicted = 70% - 80% = -10%; absolute value is 10%). The Proficiency Prediction Score is calculated as follows: Proficiency Prediction Score = 100% - Error Rate In this example, the score is as follows: Proficiency Prediction Score = 100% - 10% = 90% A higher Proficiency Prediction Score indicates a larger number or percentage of correct proficiency predictions. In this example, Discovery Education Assessment had a score of 90%. Discovery Education Assessment uses information from these scores to improve its benchmark assessments every year. Discovery Education Assessment Proficiency Predictions vs. NM SBA Proficiency Levels In the school year, districts in New Mexico used the Discovery Education Common Core interim benchmark assessments to predict performance on the NM Standards Based Assessments (SBA). The following tables display the DEA percent Proficient on the Common Core interim benchmark assessments, percent Proficient on the New Mexico SBA, the difference between the two and the proficiency prediction score. The median reading proficiency prediction score was 98.1 while the median math proficiency prediction score was Validity: New Mexico Reading Proficiency Prediction Scores from DEA CC NM SBA Difference Prediction Score Grade Grade Grade Grade Grade Grade Median Page 28

31 Validity: New Mexico Math Proficiency Prediction Scores from DEA CC NM SBA Difference Prediction Score Grade Grade Grade Grade Grade Grade Median New Mexico Reading % Proficient: DEA Common Core & SBA DEA CC NM SBA Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 70 New Mexico Math % Proficient: DEA Common Core & SBA DEA CC NM SBA 10 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Page 29

32 Vertical Scale Averages and Growth Growth models depend on a highly rigorous and valid vertical scale to measure student performance over time. Discovery Education Assessment vertical scales are constructed using Rasch measurement models with state-of-the-art psychometric techniques. The accurate measurement of student achievement over time is becoming increasingly important to parents, teachers, and school administrators. Student growth within a grade and across grades has also been sanctioned by the U. S. Department of Education as a reliable way to measure student proficiency in Reading and Mathematics and to satisfy the requirements of Adequate Yearly Progress (AYP) under the No Child Left Behind Act. Accurate measurement and recording of individual student achievement can also help with issues of student mobility: as students move within a district or state, records of individual student achievement can help new schools administer to the needs of this mobile population. The assessment of student achievement over time is even more important with the use of benchmarks tests. Discovery Education Assessment Benchmark tests provide a snapshot of student progress toward state standards at up to four points during the school year. These benchmark tests are scientifically linked, so that the reporting of student proficiency levels is both reliable and valid. Discovery Education Assessment has added a scientifically based vertical scaled growth score to its family of benchmark tests in These growth scores are based on the Rasch measurement model, a state-of-the-art psychometric technique for scaling ability (e.g., Wright & Stone, 1979; Wright & Masters, 1982; Linacre 1999; Smith & Smith, 2004; Wilson, 2005). To accomplish vertical scaling, common items are embedded across assessments to enable the psychometric linking of tests at different points in time. For example, a Grade 3 mathematics benchmark test administered mid-year might contain below grade level and above grade level items. Performance on these off grade level items provides an accurate measurement of how much growth occurs across grades. Furthermore, benchmark tests within a grade are also linked with common items, once again to assess change at different points in time within a grade. Discovery Education Assessment is using established psychometric procedures to build calibrated item banks and linked tests (i.e., Ingebo, 1997; Kolen & Brennan, 2004). Isn t student growth similar across grades? Don t students change as much from Grade 3 to Grade 4 as they do from Grade 7 to Grade 8? Previous research on the use of vertical scales has demonstrated that student growth is not linear; that is, growth in student achievement is different from grade to grade (see Young 2006). For instance, the figure on the next page shows preliminary Discovery Education Assessment vertically scaled growth results. This graph shows growth from Grades Kindergarten to 10 in Mathematics as measured by Discovery Education Assessment s Spring benchmark tests. Typically, students have larger gains in mathematics achievement in elementary grades with growth somewhat slowing in middle and high school, as published by other major testing companies. Page 30

33 Average Scale Score Avg Student Scale Score National Math Scale Score Averages Kindergarten Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Student growth can now be accurately measured at four points in time in each grade level. Discovery Education Assessment benchmark tests are administered up to four times yearly: Early Fall, Late Fall, Winter, and Spring. For each time period, we report scale scores and accompanying statistics. Most testing companies only allow the measurement of student growth at two points in time: Fall and Spring. Discovery Education Assessment benchmark tests provide normative information to assess student growth multiple times each year. The figure to the right illustrates this growth for Grade 4 Mathematics using our benchmark assessments. Discovery Within Year Growth for 4th Grade Math Math Test P Test A Test B Test C Page 31

34 Common Core Average Vertical Scale Scores N Test A Test B Test C Average Average Average Scale Scale Scale Score St. Dev. N Score St. Dev. N Score St. Dev. Kindergarten 44, , , Grade 1 83, , , Grade 2 77, , , Grade 3 55, , , Grade 4 59, , , Grade 5 59, , , Grade 6 58, , , Grade 7 56, , , Grade 8 55, , , English 1 16, , English 2 11, , N Test A Test B Test C Average Average Average Scale Scale Scale Score St. Dev. N Score St. Dev. N Score St. Dev. Kindergarten 43, , , Grade 1 82, , , Grade 2 77, , , Grade 3 55, , , Grade 4 59, , , Grade 5 59, , , Grade 6 58, , , Grade 7 56, , , Grade 8 54, , , Algebra 1 11, , Algebra 2 12, , Geometry 10, , Page 32

35 Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test A Test B Test A Test B Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test C Test A Test B Test A Test B Comon Core 1213 Reading Avg Scale Scores K E1 E2 Grade Common Core 1213 Math Avg Scale Scores K A1 A1 GE Grade Page 33

36 Appendix A: Test and Question Statistics, Reliability, and Scale Scores The following section reports test and question statistics, reliability, and percentiles for the benchmark tests, for grades 3-8, Reading and Mathematics. These benchmark tests were administered during the fall of Benchmark tests are revised each year based on test and question statistics. Number of Students: Number of Items: Mean: Standard Deviation: Reliability: SEM: Scale Score: Level Question P-values: Biserial: Rasch Item Difficulty: DIF Gender: DIF Ethnicity: Number of students used for calculation of test statistics. Number of items in each benchmark test (including common items used for scaling purposes). Test mean in terms of number correct. Test standard deviation. Cronbach s alpha. Standard Error of Measurement (SEM) for the test. Discovery Education Assessment Scale Score for each number correct (Scale scores are vertically scaled using Rasch measurement. Scale scores from grades K-12 range from 1000 to 2000). The DEA proficiency level (Level 1 Level 4) assigned to the student based on the number of items correct on the assessment. The proportion correct for each item. Item discrimination using biserial correlation. Rasch item difficulty parameter calculated using WINSTEPS. Rasch item difficulty difference (Male vs. Female). Rasch item difficulty difference (White vs. Black). DIF Size Negligible: Moderate: Large: 0 logits to.42 logits (absolute value)..43 logits to.63 logits (absolute value)..64 logits and up (absolute value). (see p.1070 An Adjustment for Sample Size in DIF Analysis, Rasch Measurement Transactions, 20:3, Winter 2006) Page 34

37 Technical Data Common Core Fall Reading Grade 3 Test Statistics Number of Students 13,163 Number of Items 34 Average Number Correct Std. Deviation 6.74 Avg. Scale Score 1401 Reliability 0.86 Std. Error of Measurement 2.52 Question Statistics Rasch Item Item No. P-Value Biserial Difficulty DIF Gender DIF Ethnicity Scale Scores & Percentiles No. Correct Scale Score Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level Level 4 Level Level 4