2014 Texas Public School Rankings Methodology April 2014

Transcription

1 2014 Texas Public School Rankings Methodology April 2014 Robert Sanborn, Ed.D. Caroline Neary, M.Ed.

2 School Rankings 2014 pg. 1 I. Introduction A. About CHILDREN AT RISK CHILDREN AT RISK is a 501(c)(3) non-profit, non-partisan research and advocacy organization dedicated to addressing the root causes of poor public policies affecting children. The organization began in the fall of 1989 when a group of child advocates met to discuss the lack of documentation on the status of children and the absence of strong public policy support for youth. Since then, the organization has produced eleven major publications including Growing Up in Houston, which focuses on critical children s issues and is published biennially. Over the course of two decades, CHILDREN AT RISK has evolved from an organization researching the multitude of obstacles our children face to one that also drives macro-level change to better the future of our city and state through community education, collaborative action, evidencebased public policy, and advocating for our youth at the local and state level. Through its Public Policy and Law Center established in 2006 as the only center of its kind in Texas CHILDREN AT RISK uses policy and legal expertise as a powerful tool to drive change and create a better future for our children. In recent years, CHILDREN AT RISK has grown exponentially in its capacity to speak out and drive change for children and has become a premier resource on children s issues among major media outlets, public officials, and the non-profit sector. Today, the mission of CHILDREN AT RISK is to improve the quality of life for children across Texas through strategic research, public policy analysis, education, collaboration, and advocacy. Since 2006, the school ranking system developed by CHILDREN AT RISK has highlighted the successes and need for improvement of local public schools. As a research and advocacy organization, the purpose of the rankings is not only to provide a tool to parents and students, but also to provide information to campuses and districts on how they perform relative to their peers and compared against successful models of high-performing public schools. In 2009, CHILDREN AT RISK began to include all eligible high schools in the state of Texas and extended the ranking system to include eligible elementary and middle school campuses. Thus far, the CHILDREN AT RISK rankings have proven to be instrumental in generating conversations among educators and the public regarding methods for improving our public education system. In addition, the School Rankings aim to: Serve as an accessible guide for parents, educators, and community members on the performance of local schools. Generate conversations not just about the data used in the ranking, but around how schools and districts are performing overall in creating college-ready students. Be transparent. Research is strongest when it is made available to the public and open to scrutiny. Thus, discussion can be generated, the ranking methodology can be improved upon, and all districts can utilize this avenue of assessing campuses. Encourage the use of data in public school reform. The rankings have successfully encouraged further data analysis at the campus and district level, targeted school intervention, aided teacher and staff professional development, allocated funds to better serve children, and promoted changes in strategic planning.

3 School Rankings 2014 pg. 2 Each year, CHILDREN AT RISK reexamines its methodology of ranking schools to ensure that this report most accurately reflects school performance, utilizes the best data available, and incorporates feedback from educators, researchers, and service providers. Because CHILDREN AT RISK continually improves the methodology, this year s School Rankings are not directly comparable to previous years results. B. Collaboration Portions of the data analysis for the 2014 School Rankings were done in conjunction with Education Resource Group and Dr. Lori Taylor of Texas A&M University. II. Methods A. School Ranking Methodology Campuses are ranked across three indexes: Student Achievement, Campus Performance, and Growth. Within each index, a percentile rank was calculated for each campus. Using these three index percentiles, a weighted average was computed to create an overall composite index. A state or local rank was determined as the order in which campuses are listed when the weighted Composite Indexes are sorted from highest to lowest, relative to other schools servicing the same grade levels (i.e., Elementary, Middle, High). B. Indexes Student Achievement Index The Student Achievement Index reflects raw performance in key achievement areas. The Student Achievement indicators included and their weights are as follows: ELEMENTARY AND MIDDLE SCHOOL ACHIEVEMENT INDICATORS STAAR Reading- Advanced 50% STAAR Math- Advanced 50% HIGH SCHOOL ACHIVEMENT INDICATORS STAAR Reading- Advanced 30% STAAR Math- Advanced 30% C@R Graduation Rate 40% For each indicator (e.g., STAAR Reading- Advanced), campuses were ordered highest to lowest by their score and a percentile rank was calculated. The percentile rank indicates the

4 School Rankings 2014 pg. 1 percentage of scores that fall at or below that score. Each indicator has a pre-determined weight and the weighted average of these percentiles became the Student Achievement Index. For more detailed definitions of achievement indicators, please see Section IV. Campus Performance Index The Campus Performance Index captures performance on the Student Achievement indicators using demographically adjusted values. Raw academic measurements, such as those in the Student Achievement Index, have a bias toward campuses with a low percentage of economically disadvantaged students. The Performance Index is created to measure the effectiveness of the people and programs at a campus independent of the differences in student demographics. Linear regression analysis demonstrates the relationship between the percent of economically disadvantaged students and their performance on achievement indicators. Deviation from expected value is defined as the difference between the actual pass rate of a campus and its forecasted performance as defined by the regression line in this analysis. Each campus received a positive or negative deviation score, based on the difference between its actual score and the expected score. For each indicator (e.g., STAAR Reading- Advanced), campuses were ordered highest to lowest by their deviation score and a percentile rank was calculated. The percentile rank indicates the percentage of scores that fall at or below that score. Each indicator has a pre-determined weight and the weighted average of these percentiles became the Campus Performance Index. Growth Index The Growth Index captures improvement over time in standardized testing. The Growth Index is composed of gain scores in math and reading, which measure student-level performance from one year to the next, relative to others with the same past score in the subject. For example, 2013 math scores for all tenth-grade students with a given ninth-grade math score were compared to their ninth grade math scores in The year-to-year difference was standardized and transformed into a normal curve equivalent score for ease of interpretation. The campus level scores are averages of the student level normal curve equivalent gains from one year to the next, where a student with an average gain among students who had the same prior score as she did would have a normal curve equivalent gain of 50. The exams included in the gain score calculations include: STAAR Reading, English I EOC, English II EOC, TAKS Reading/ELA, STAAR Math, Algebra I EOC, Geometry EOC, Algebra II EOC, and TAKS Math. For each indicator (e.g., Reading gain score), campuses were ordered highest to lowest by their score and a percentile rank was calculated. The percentile rank indicates the percentage of scores that fall at or below that score. The average of the two percentiles became the Growth Index.

5 School Rankings 2014 pg. 2 C. Letter Grades All campuses were assigned a letter grade, based on their Composite Index. For elementary, middle, and high schools: Campuses at or above the 75 th percentile (indicating they rank better than at least 75% of schools at that grade level) received an A. Campuses at or above the 55 th percentile, but below the 75 th percentile, received a B. Campuses at or above the 35 th percentile, but below the 55 th percentile, received a C. Campuses at or above the 15 th percentile, but below the 35 th percentile, received a D. Campuses below the 15 th percentile received an F. Once campuses were assigned a general letter grade, A, B, and C grades were further differentiated into plus/minus grades. The range of Composite Index scores for each letter grade was divided evenly into thirds. The top third of Composite Index scores became plus grades and the bottom third of Composite Index scores became minus grades. The cut-points are different for elementary, middle and high schools because they are based on the unique sample of scores for this year s schools at each level. Letter Grade ELEMENTARY MIDDLE HIGH Minimum Percentile Minimum Percentile Minimum Percentile A A A B B B C C C D F III. Data and Limitations A. Overview of School Rankings Data To rank public schools across Texas, CHILDREN AT RISK compiles and analyzes data collected by the Texas Education Agency (TEA) through the Texas Academic Performance Reports system (formerly the Academic Excellence Indicator System) and direct requests to TEA. CHILDREN AT

6 School Rankings 2014 pg. 3 RISK emphasizes utilizing an array of indicators which encourages a holistic examination of school quality when evaluating campuses. CHILDREN AT RISK seeks to hold schools accountable for student performance on standardized testing in addition to other measures such as graduation rates and improvement over time. The included indicators and weights applied to each indicator in the CHILDREN AT RISK ranking calculation were determined by staff members and influential members of the education community. More weight is given to indicators that better predict college success based on a growing body of research. Other prominent school ranking systems (e.g., Newsweek and Philadelphia Magazine) apply a similar methodology where more weight is given to indicators that are more important and that come from more objective data sources with fewer missing data points. B. Missing Data and Excluded Schools For a school to be included in the school rankings, a campus must have complete data from the Texas Education Agency (TEA) for each of the indicators included in the analysis. If a high school campus was missing graduation rate data for the most recent year, the previous year s graduation rate was used if possible. If a campus was missing gain score data for the most recent year, its STAAR- Proficient (percentage of students scoring at Final Level II or above on the STAAR exam) percentile was substituted. Any campus that was ultimately missing one or more data points was excluded from the analysis. Campuses that are under an alternative accountability system from TEA (i.e., disciplinary sites) and campuses with fewer than 100 students enrolled were excluded from the rankings. Campuses confirmed to be undergoing a state or district investigation were also excluded. Elementary school campuses must contain at least two grades from Kindergarten-5th; middle school campuses must include at least two grades from 6th-8th; high school campuses must have a graduating 12 th grade class. Campuses that serve elementary, middle, and high school students, or two of these three groups, are evaluated in each ranking for which they qualify. C. District Rankings Districts were ranked against the same criteria as campuses. Charter districts and districts with fewer than 100 students enrolled were excluded. Large districts were defined as those with 30,000 or more students enrolled. The 4-year Graduation Rate reported by the Texas Education Agency was used in place of the CHILDREN AT RISK Graduation Rate. Like campuses, districts were assigned a general letter grade based on their composite index; however, districts were not further differentiated into plus/minus grades. D. Geographic Sub-Lists CHILDREN AT RISK rankings are computed at the elementary, middle, and high school levels across the state of Texas. The School Rankings analysis is conducted at the state level before campuses are extracted to rank schools in smaller geographic areas (i.e., Houston, North Texas, Austin, and San Antonio).The Greater Houston area is defined as the following eight counties: Brazoria, Chambers, Fort Bend, Galveston, Harris, Liberty, Montgomery, and Waller. The North

7 School Rankings 2014 pg. 4 Texas area includes the following nine counties: Collin, Dallas, Denton, Ellis, Hunt, Johnson, Kaufman, Rockwall, and Tarrant. The Greater Austin area includes seven counties: Bastrop, Blanco, Burnet, Caldwell, Hays, Travis, and Williamson. Finally, Greater San Antonio is defined as the following six counties: Atascosa, Bandera, Bexar, Comal, Guadalupe, and Medina. E. Explanation of Sub-Lists Peer Lists: These lists are composed of schools that meet certain criteria along key socioeconomic indicators. They are intended to show how schools stack up against their peers - campuses that are similar in terms of student populations. The indicators used to determine peers are: Category Indicator Cut-off PEER RANKINGS INDICATORS Poverty High % Economically Disadvantaged 74.5% - High Schools 84.2% - Middle Schools 91.4% - Elementary Schools Minority Populations High % Hispanic 69.6% - High Schools 75.3% - Middle Schools 81.6% - Elementary Schools Minority Populations High % Black 24.5% - High Schools 28.2% - Middle Schools 29.3% - Elementary Schools High designations of poverty and minority populations were based on statistics for the sample of schools included in the school rankings. For each indicator, the mean and standard deviation were calculated for included schools at each level. Schools were placed in the high range if their data point for the specific indicator fell at least one standard deviation above the population mean. Separate cut-offs were calculated for high schools, middle schools, and elementary schools. Gold Ribbon Schools: The top schools that house a high concentration of economically disadvantaged students (based on the Economically Disadvantaged Peer List). F. Study Limitations There are numerous factors that affect the success of children and schools. Research shows some of the biggest factors for student success are parental involvement, social and emotional development, participation in extracurricular activities, teacher and parent expectations of students, and engaging class work that stimulates critical thinking. However, there is no standard measure for any of these constructs, and it would be particularly difficult to collect these data efficiently and consistently for ranking over 7,000 schools. Another constraint is CHILDREN AT RISK s dependence on data collected by the Texas Education Agency (TEA). Thus, the limitations posed by TEA data are valid criticisms for this school ranking system. Any erroneous data reported to or by TEA may have an effect on the rankings. This includes the gain

8 School Rankings 2014 pg. 5 score data; CHILDREN AT RISK has no way to verify the veracity of reported gains and must trust schools reporting. Additionally, the CHILDREN AT RISK ranking is limited to campuses that have complete data available through TEA for all measures included in the ranking. IV. Achievement Indicator Definitions A. Elementary and Middle School Indicators STAAR Reading- Advanced: The percentage of students scoring at Level III Advanced on the STAAR Reading exam (sum of all grades tested). This standard indicates that students are well prepared for postsecondary success. STAAR Math- Advanced: The percentage of students scoring at Level III Advanced on the STAAR Math exam (sum of all grades tested). This standard indicates that students are well prepared for postsecondary success. B. High School Indicators STAAR Reading- Advanced: The percentage of students scoring at Level III Advanced on the STAAR Reading exams (sum of all grades tested). This standard indicates that students are well prepared for postsecondary success. STAAR Math- Advanced: The percentage of students scoring at Level III Advanced on the STAAR Math exams (sum of all grades tested). This standard indicates that students are well prepared for postsecondary success. Graduation Rate: The four-, five-, or six-year graduation rate calculated by CHILDREN AT RISK. Tracking first-time freshmen entering in , , and and following them to any Texas public high school for six, five, or four years, respectively, this rate is the percentage of students that have graduated by 2012 from any Texas public high school. Any students who passed away or began home schooling within four, five, or six years of beginning high school were removed from the first-time freshman cohort. The cohort of first-time freshmen used in this calculation is built by the Texas Education Agency using the same rules the Agency applies for the cohort used for its own Graduation, Dropout, and Completion Rates for state and federal accountability purposes. First-time freshmen are those students enrolled in ninth grade for the first time, looking back at five years of enrollment data in the Public Education Information Management System (PEIMS), who are attributed to the campus which he or she most recently attended. For each campus, the highest value among the four-, five-, and six-year graduation rate calculation was selected to include in the Rankings analysis. This method does not account for the small percentage of students who transfer to private school or leave the state/country after having started high school as freshman. Due to TEA s need to protect student privacy, some student level data was hidden, or masked. Schools were given

9 School Rankings 2014 pg. 6 the maximum benefit of the doubt when filling in masked data (a 4 was substituted for missing total graduate numbers, total number of students who began home schooling, or for total number of student deaths; a 1 was substituted for missing first-time freshmen cohort numbers). At times, these substitutions resulted in a graduation rate greater than 100%. In these cases, a school s final graduation rate was rounded down, and they were credited with a 100% graduation rate.

10 School Rankings 2014 pg. 7 Appendix A Tables of Indicators for Public School Rankings ELEMENTARY AND MIDDLE SCHOOLS Student Achievement Index STAAR Reading- Advanced 50% STAAR Math- Advanced 50% Campus Performance Index STAAR Reading- Advanced (Demographically Adjusted) 50% STAAR Math- Advanced (Demographically Adjusted) 50% Growth Index Reading Gain Score 50% Math Gain Score 50% 60% 20% 20% HIGH SCHOOLS Student Achievement Index STAAR Reading- Advanced 30% STAAR Math- Advanced 30% C@R Graduation Rate 40% Campus Performance Index STAAR Reading- Advanced (Demographically Adjusted) 30% STAAR Math- Advanced (Demographically Adjusted) 30% C@R Graduation Rate (Demographically Adjusted) 40% Growth Index Reading Gain Score 50% Math Gain Score 50% 60% 20% 20%