Assessment Instrument Table: STAR Math

Transcription

1 Assessment Instrument Table: STAR Math Element Description Assessment Instrument Information Instrument Name Name of specific instrument (more than vendor name). STAR Math Vendor Purpose (Intended Use) Population When? How frequently? Content Area (s) Learning Objectives Name of the company or organization that produces the instrument. The described purpose and appropriate uses of the instrument. Who (which students) could be assessed using the instrument. How frequently the instrument can be administered in a school year, and recommended or required administration windows. Content area or areas being assessed. Specific learning objectives being assessed, at as detailed a level as is provided. This may be "topics" or categories or may be actual learning objective statements. Renaissance Learning, Inc. STAR Math is a student-based, computer adaptive assessment for measuring student achievement in math. STAR fulfills a variety of assessment purposes, including interim assessment, screening, standards benchmarking, skills-based reporting and instructional planning, and progress monitoring. Students in grades 1 through 12 As an interim assessment, STAR was designed for frequent administration. The STAR Math assessments fit virtually any assessment schedule with minimal impact on instructional time and administrative workload. Educators can administer STAR three times per year in fall, winter, and spring. Educators may also administer STAR as a progress monitoring assessment as often as weekly. Math STAR Math is a standards-based test that uses items that measure standards appropriate to a student s grade, or standards the student should have mastered at lower grades. STAR Math assesses the following key domains of the CCSS for mathematics: Numbers & Operations; Data Analysis, Statistics, & Probability; Algebra; and Geometry & Measurement. For a list of specific skills assessed within each domain, please see the table below. Page 1

2 Numbers & Operations Data Analysis, Statistics, & Probability Algebra Geometry & Measurement Count with objects & numbers Identify odd and even numbers Relate place & value to a whole number Add & subtract whole numbers without and with regrouping Multiply whole numbers Divide whole numbers without a remainder in the quotient Divide whole numbers with a remainder in the quotient Identify, compare, & order fractions Add & subtract fractions with like denominators Find prime factors, common factors, & common multiples Add & subtract fractions with unlike denominators Convert between an improper fraction & a mixed number Relate a decimal to a fraction Relate place & value to a decimal number Add or subtract decimal numbers Divide a whole number resulting in a decimal quotient Multiply & divide with Read or answer a question about charts, tables, or graphs Use a chart, table, or graph to represent data measure of central tendency Use a proportion to make an estimate Determine the probability of one or more events Relate a rule to a pattern Determine the operation given a situation Graph on a coordinate plane Evaluate an algebraic expression or function Solve a linear equation linear equation Identify characteristics of a linear equation or function Solve a system of linear equations system of linear equations Simplify an algebraic expression Solve a linear inequality Solve a nonlinear equation Graph a 1- Relate money to symbols, words, & amounts Use the vocabulary of geometry & measurement missing figure in a pattern measurement Tell time Calculate elapsed time Solve a problem involving the perimeter of a shape Solve a problem involving the area of a shape Identify congruence & similarity of geometric shapes Solve a problem involving the surface area or volume of a solid missing measure or dimension of Page 2

3 fractions Relate a decimal number to a percent Solve a proportion, rate, or ratio Evaluate a numerical expression Perform operations with integers square root Solve a problem involving percentages Individual Metrics Individual Comparison Points (cut scores) The scores provided at the individual (student) level. Information provided regarding how good is good enough performance on the instrument. Comparison variable inequality a shape Scaled score (SS) is useful in comparing student performance over time and in identifying student performance in relation to a vertical scale and all criterion and norms associated with that scale. The STAR Math scaled scores range from Grade equivalent (GE) score, ranging 0.0 to 12.9+, is norm-referenced and represents how a student s test performance compares with other students nationally. Normal curve equivalent (NCE) score, ranging from 1 99, is norm-referenced score similar to the percentile rank score but based on an equal-interval scale. This means the difference between any two successive scores on the NCE scale has the same meaning throughout the scale. Percentile rank (PR) score, ranging from 1 99, is a norm-referenced score that provides the best measure of a student s level of math achievement compared to other students in the same grade nationally. The score indicates the percentage of a student s peers whose scores were equal to or lower than the score of that student. Student growth percentile (SGP) is a measure of growth between a pre- and post-test relative to the growth made by other students in the same grade with the same pre-test score. It is a simple and effective way for educators to interpret a student s growth rate relative to that of his or her academic peers nationwide. Algebra readiness indicator is a score that helps teachers identify student progress through foundational math skills and ensure that they are on track to be ready for algebra. Scaled score (SS) all the results of STAR Math tests across grade levels are converted to a common scale using an IRT model; these scores range from 0 to These Scaled Scores are useful in comparing student performance over time and in identifying performance and all criterion and norms associated with that scale. Page 3

4 information should be The following scores which include comparison points in the score are also provided: available for every Grade equivalent (GE) score indicates the grade placement of students for whom a particular score individual metric. This may be performance is typical. If a student receives a GE of 10.7, this means that the student scored as well on STAR level ratings with Math as did the typical student in the seventh month of grade 10. Ranging 0.0 to 12.9+, GE scores specific cut scores. represent how a student s test performance compares with other students nationally, they are norm referenced. Percentile rank (PR) score, ranging from 1 99, is a norm referenced score that indicates the percentage of a student s peers whose scores were equal to or lower than the score of that student. Percentile Ranks show how an individual student s performance compares to that of his or her same-grade peers on the national level. Normal curve equivalent (NCE) score, ranging from 1 99, is norm referenced. It is similar to the percentile rank score but is based on an equal-interval scale; NCEs are scaled in such a way that they have a normal distribution, with a mean of 50 and a standard deviation of in the normative sample for a given test. Student growth percentile (SGP) is a measure of growth between a pre- and post-test relative to the growth made by other students in the same grade with the same pre-test score. It is a simple and effective way for educators to interpret a student s growth rate relative to that of his or her academic peers nationwide. SGPs for STAR Math Enterprise are calculated using an approach similar Aggregate Metrics Scores provided at the group level. The groups for which scores are reported. Note: the group could be a grade level, school, district, or disaggregated groups (e.g. race/ethnicity, gender, IEP status, FRL status) Specify the group(s) and the score(s) provided. to the Colorado Growth Model. When districts set cut scores for individual student scaled scores to establish performance categories, these categories are used to provide aggregate metrics, including the percent and number of students by district benchmark category (available by grade at the district, school levels) across years of available data. These metrics can be calculated using cross-sectional data (same grade year to year) or for the same students over multiple years. [Longitudinal Report] The following additional aggregate metrics are also provided: Median Student Growth Percentile -- the middle student growth percentile within the included group. This metric is reported for different time periods (fall to spring, spring to spring) by grade level within school, grade level within the district, and by class.[growth Report, Student Growth Percentile Report] Average scores at the school by grade level and classroom levels of the following individual metrics: Scale Score, Grade Equivalent, Percentile Rank, and Norm Curve Equivalent. Percent of students in or above the estimated mastery range for math standards (CCSS or CAS) by Page 4

5 school and by class within the school - STAR Math provides an estimate of the students' mastery of standards by aligning them to the same 1400-point difficulty scale used to report STAR scores. The Estimated Mastery Range identifies a band of scores where the student is just below or above mastery. The percentage of students who score in or above this range indicates overall progress toward standards mastery. [State Standards Report] Aggregate Comparison Points (vendor) Aggregate Comparison Points (CDE) Information provided regarding how good is good enough performance at the group level. Cut points established for requests to reconsider. Because most of the individual metrics provided for Star Math are norm-referenced, almost all of the aggregate metrics also include a comparison point within the metric definition. This includes the following metrics (described above): Percent/Number Scoring at District-determined Performance Levels -- note districts determine the comparison points used in these metrics when they set their own cut scores for different performance levels. Median Growth Percentiles Average Grade Equivalent Average Percentile Rank Average Norm Curve Equivalent Percent of Students in or Above Estimated Mastery Range for Math Standards See tables below for aggregate metrics for which CDE established comparison points as part of the Request-to-Reconsider process. CDE provided comparison points include Fall and Spring Mean Scale Scores and Median Growth Percentiles for each grade level. Note: The CDE comparison points for STAR Math for the Request-to-Reconsider process are currently being revised. Grade Fall Scale Scores 50th Percentile Mean Scale Score Mathematics Scale Scores by Grade Level Spring Scale Scores 50th Percentile Mean Scale Score Scale Score Growth (Fall to Spring) Median Growth Percentile Meets Level Page 5

6 Data Reports Alignment Description of data reports that are provided/available at the individual and aggregate level(s). Information provided by the vendor about alignment of this instrument to other instruments, standards, etc. Below we describe some key STAR Math Reports, including the levels for which the report is available. Growth Reports and Growth Proficiency Charts (student, class, grade, school, district) show educators whether students are reaching their growth expectations. The Growth Report includes student growth percentile (SGP) and calculates the change between two STAR test scores. The Growth Proficiency Chart plots SGP and proficiency on a quadrant graph so that educators can easily see whether students are challenged and growing every year, regardless of their academic starting point. Longitudinal Reports (grade, school, district) show educators whether students in each grade are growing from year to year. Educators can compare the same grade year to year or the same students over multiple years. State Performance Reports (student, class, grade, district) predict student performance on highstakes tests. Predictions account for the amount of growth that typically occurs between the date of the last STAR test taken and the date of the state test. State Standards Reports (student, class, grade, district) gauge students current and projected mastery according to the Colorado Academic Standards as well as the Common Core State Standards. Users may choose which of the standards to view on these reports. Samples of additional data reports and a detailed explanation of each report s purpose(s) can be viewed at: The vendor provides an alignment report which illustrates how the specific skills assessed by STAR Math map to the Colorado Academic Standards including Level Expectations (GLEs) and Evidence Outcomes (EOs). See for the alignment report for grades pre-k 12. The report lists the grade, standard, grade-level expectations (GLE), and high school expectations (along with the evidence outcome) together with the skills assessed within STAR Math that are aligned to the Colorado Academic Standards. Note that STAR Math is aligned to Standards 1 4 of the Colorado Academic Mathematics Standards. STAR Math skills include alignments under each grade-level expectation within these standards for grades pre-k 12. Page 6

7 The vendor provides a similar alignment report illustrating how the skills assessed by STAR Math map to the Common Core State Standards (CCSS). See for this alignment report. During the analysis of the alignment between STAR Math skills assessed to the CCSS, two types of alignment emerged and serve to clarify the way in which product skills meet standards. The alignment type was either direct or foundational. The CCSS the alignment report includes the following types of alignments: Direct alignment the product skill directly and specifically meets one or more of the skills in the standard either fully or partially. Foundational alignment the product skill is necessary to meet the skill in the standard either through a prerequisite or related relationship. Technical Quality Information about the technical quality of the instrument. Reference to technical analysis if available electronically. The STAR Math Technical Manual (available at R FEC4F604.pdf) presents details about the assessment s technical quality. Key information is summarized below. Content Validity STAR Math s content validity is based on expert analyses of state and college- and career-ready standards, curriculum materials, test frameworks, and content-area research, including best practices for mathematics instruction. Content-related validity evidence is presented in the technical manual in the extensive treatments of the content of the item banks; the rationale and research basis for selecting that content; and the methodologies employed to develop, evaluate, and select test items. Content-related validity evidence is also presented in the discussion of the methods used to align skills and items to a wide variety of standards, including but not limited to state curriculum standards, the Common Core State Standards, and college- and career-readiness standards. Reliability Reliability is the extent to which a test yields consistent results from one administration to another and from one test form to another. The question of the reliability of the test was approached in two ways: by calculating split-half reliability and by calculating test-retest reliability, in both cases for both Scaled Scores and Standardized Scores. STAR Math s reliability is high. Reliability for all grades combined is 0.95; at each grade from 1 12 it ranges from Split-half reliability for all grades combined is about 0.94; for each grade from 1 12 it ranges from Alternate-form reliability for all grades combined is about 0.91; at each grade from 1 12 it ranges from Validity Renaissance Learning provided two types of data to demonstrate the construct validity of STAR Page 7

8 Math. This included data that demonstrated a strong and positive correlation between STAR Math scores and scores on other standardized tests, and data showing a strong degree of relationship between STAR Math scores and teacher ratings of their students proficiency in selected math skills. Meta-analysis of the STAR Math validity data (i.e. correlations between STAR Math scores and scores on other standardized tests) using 506 correlation coefficients resulted in an overall estimate of the validity of STAR Math of 0.71, with a standard error of See the Technical Manual for a list of assessments included in this meta-analysis and a discussion of the relationship between STAR Math scores and teacher ratings of their students proficiency in math skills. Norms The test's norms are based on a 2012 norming study. The vendor used a multi-stage stratified random sampling process to ensure a sample nationally representative of geographic regions, metropolitan classification codes, different school sizes and socioeconomic status classifications, and score ranges within each grade from last three years of STAR Math data (fall 2008 spring 2009, fall 2009 spring 2010, and fall 2010 spring 2011). Grade Equivalent (GE) scores for each grade and each month of the school year were computed using the median STAR Math scaled scores for each grade and month. Scaled Score to Grade Equivalent and to Percentile Rank conversion tables for the empirical norming period are provided in the technical manual. This norming approach allows STAR Math to provide normative information that is most relevant, regardless of the specific time period in which schools administer the STAR Math test to students. Page 8