Nevada Growth Model FAQ

Transcription

1 Q: What do those bubbles mean? A: The bubbles are Nevada s way of showing you how schools academic performance compares to each other, and to all other schools in the state on average. The higher a bubble is in the chart, the higher its percentage of proficient and advanced students in that subject area. This is called achievement level, or status. The further to the right in the chart a bubble is, the higher that school s typical growth. The chart can be roughly divided into four quadrants but two lines, one vertical and one horizontal. The colored horizontal line that appears when you are looking only at elementary, middle, or high schools is the average school achievement level across the state. Every elementary school in Nevada has a percentage of its kids that are proficient or advanced this line represents the average of all these school- level percentages, or the state total. Similarly, the vertical line at for growth percentile of 50 represents the state "average" growth for all kids (it s actually a median, not an average, but it s the same idea). So the location of a school s bubble both from left to right and from down to up tells us about growth and achievement for that school at the same time. Q: What is a scale score? A: Scale score is an examinee s calculated total score on a test. The Nevada Criterion Referenced Tests (CRT) scales used by Nevada go from points. Note that this is different from a "raw score," which would be a tally of the total number of questions on a test that an examinee answered correctly. The use of scale scores makes it easier to interpret test results in many cases. Q: Are all students with student growth percentiles included in the bubble for a district/school/other group? A: Not exactly. In order to receive a student growth percentile in a CRT content area, a student needs to have valid scores in that content area from two consecutive years, following a normal grade progression. Every student that matches these criteria will have a student growth percentile calculated, and educators who have access to the data can get those individual growth scores even when they are not represented in a bubble. The growth level (a bubble's horizontal position in the plots) for a school or any subgroup of a school includes only those students who were enrolled in that school on the count date of that academic year. The growth level (a bubble's horizontal position in the plots) for a district or any subgroup of a district is only for those students who were enrolled in a school in that district by the count date of that academic

2 year, or if they were continuously enrolled in that district for two consecutive years regardless of their count date status. Finally, bubbles are only displayed for groups that have 25 or more students with growth scores. Q: Why are some bubbles bigger than others? A: The larger a school in terms of total enrollment, the larger its bubble is. Note that total enrollment can be quite different from the number of students that take the CRT, or the number of students that have growth percentiles, or the number of students included in school or district growth calculations. Q: Some bubbles look really small - how many kids are in those bubbles? A: Public data are always protected by only showing bubbles that contain 25 or more individuals' growth scores. For that reason, you will never see a bubble that represents fewer than 25 individuals. This practice protects the privacy of those individuals so that it would be impossible to know who exactly they were through some process of elimination. Q: How do I find scores for a different content area or year? A: The square labels in the upper right- hand corner of the application window toggle between math and reading. The year pull- down menu below that enables you to switch years. The year shown is the second calendar year of a given academic year, so 2010 refers to the academic year. Q: What is the state proficiency line and how do I interpret it? A: The state proficiency line is a way of understanding how a particular school's achievement level compares to that of all other schools in the state at that level. A school whose bubble appears below the state proficiency line has a lower percentage of students scoring Proficient and Advanced than the average of all schools in the state. The line is calculated at the school level. Every elementary school in the state has a Percent Proficient/Advanced level. Those are all averaged together to get the state proficiency line level for that content area. Q: How come sometimes there is no state proficiency line?

3 A: The state proficiency line is displayed when only Elementary, Middle, or High School bubbles are displayed. It disappears when a mixture of school levels is displayed, or in the case of district or disaggregated group bubbles. In the upper right- hand area of the application window, leave only one school level (Elementary, Middle, or High School) checked, and the state proficiency line should appear. If you drill into the school's data, looking at Student Groups, you will continue to see the line corresponding to the state proficiency line for that school level. Q: How do I get demographic information about a school or district? A: When maneuvering the mouse over a bubble, a pop- up window appears giving you some growth details. In the upper right- hand corner of this "data tip" window is a blue circle with an i in it - click on it and you will get another data tip with a lot more details about the students whose data are represented in that bubble. Q: I see bubbles that represent schools. How do I show bubbles that represent districts? A: Mouse over any school bubble and choose View this school>up to Districts. All the districts currently loaded in your display will be shown. Q: I chose a district, but some of the schools in the district are grayed out in the right- hand navigation list. Why can t I see their data? A: Public data are always protected by only showing bubbles that contain 25 or more individuals' growth scores. For that reason, you can never see a bubble that represents fewer than 25 individuals. This practice protects the privacy of those individuals so that it would be impossible to know who exactly they were through some process of elimination. Q: If I check two different boxes in the Explore menu, does it find schools that match both of those criteria? A: No, in this case the application displays schools that match either of those criteria. In some cases both the criteria might be matched, but the only way to find out which ones that might be is by toggling the check boxes one at a time and keeping track of your targets.

4 Q: I would like to share what I just saw with another person by e- mail. How do I point them to the data I am looking at? A: In the Nevada Growth Model School View, you can simply copy the web address (URL) in your browser window and paste it into an e- mail address - anyone clicking on it will see the same selections that you have made. Sometimes the URL can get very long - just grab the short URL from the text box in the Share window instead, and paste that into your e- mail. Q: How can I label the bubbles in my view? Maneuvering the mouse over each one is not convenient. A: In the current version of the Nevada Growth Model, you can apply up to ten labels so that you can track those bubbles more easily. Just hit the Enable Bubble Labels button. You can move the labels around so that they appear just right for your set of bubbles. Q: How can I remove schools from the display list? A: You can only filter out a whole school level such as Middle Schools. You cannot remove one or more particular schools. Q: The link provided does not take me to a school's website, it takes me to the district's website instead. A: If a school does not supply Nevada Dept. of Education (NDE) with an up- to- date website to link to, NDE substitutes the district's website instead. This generally enables users to easily navigate to a school's website, if there is one. Q: I cannot mouse over a bubble that I want to investigate because there are too many other bubbles around it. A: If you click on the name of the school in the alphabetical list on the right, it will be easier to see exactly where the bubble is. You can further reduce the clutter by removing schools from the display that are of a level you don't need at the moment, such as Middle Schools. If you double click on the name of the school in the alphabetical list on the right, that school will be broken out by grades. If you are interested in a different breakout you can choose one from the pull- down menu in the upper left- hand corner of the application window.

5 Q: How does Nevada s growth model work? Nevada Growth Model A: The Nevada Growth Model takes a student s score in year 1 and finds all other students in the state who got exactly that same score, say in math. Then we look at all of the year 2 math scores for the same group of students and see how the student in which we are interested scored compared to the other students in the group. Growth is reported in percentiles. Q: How much growth is good enough? A: We classify student growth into three bands: High Growth is from 61st to the 99th percentile, Typical Growth is from 40th to the 60th percentile, Low Growth is from 1st to the 39th percentile. Q: When we are comparing students, to whom does the model compare my student? A: The comparison group or academic peer group consists of ALL students with that same Year 1 scale score in that grade level and for that subject we look only at test performance (academic mastery) and not at any other variable. Q: What test data is being be used to calculate growth? A: Every student with both spring 2010 and spring 2011 scores will have a growth calculation. To the extent a student has scores from fall 2008 and 2009, those scores will also be included in the growth estimates. Q: What average score is used in determining the 50th percentile at building level, district, and state? A: The average is for all Nevada students in the same grade level and content area with that particular scale score, or the academic peer group. Q: Are we comparing spring to spring data at this time? A: Yes. The release is based on the spring tests.

6 Q: Will growth model data be a part AYP? Nevada Growth Model A: Not this year. Nevada s growth model has not yet been approved for AYP use. Q: What two years of data will be included? A: Any student with spring 2010 and spring 2011 scores will be included in growth calculations, but the program will use up to the most recent four years of data. Q: How can we get questions answered? A: Questions can be ed to Q: How do you measure growth when you are at 99%? A: Any student can theoretically have growth at the 99th percentile since it would be based on performance of that student compared to others like him/her. Q: Were there any other conditioning factors for growth, other than having the same starting achievement? A: No. Only the year 1 scale score is used; all students in that grade with the same score are a part of the academic peer group. Q: Are the student peer groups pulled from the state or the district? A: State level data for that grade level, and content area is used to construct the academic peer groups. Q: Clarification: Does growth only contain two years of performance information? I know some models use as many years as are available. A: This model will use up to four years of data where available.

7 Q: So, if a student has 4 data points - 2 at one school and 2 at another, where do the scores go? A: At the school level, growth will always (only) be aggregated to the school where the student enrolled in the current school year. The individual student growth will be visible at the school of enrollment. Q: What is a growth model? A: For K- 12 education, the phrase growth model describes a method of measuring individual student progress on statewide assessments (tests) by tracking the scores of the same students from one year to the next. Traditional student assessment reports tell you about a student s achievement, whereas growth reports tell you how much change or growth there has been in achievement from year to year. Q: What questions can a growth model help answer? A: The growth model allows districts and schools to more easily identify promising, or potentially struggling, programs and practices and therefore to look deeper into what may or may not be working. A growth model can help answer such questions as: a. How much academic progress did an individual or group of students make in one or more years? b. How does an individual student s growth compare to that of students with similar prior Nevada criterion referenced test scores? c. Is a student s, school s or district s growth higher than, or lower than, or similar to typical growth? d. Which schools or districts demonstrate better than (or less than) typical growth for their students as compared to schools or districts with similar overall Nevada criterion reference test achievement? Q: Why did Nevada adopt a growth model to measure student progress? A: Annually, since 2003, Nevada has provided students, families, educators, and the general public with information about student, school and district performance based on the Nevada criterion reference test (CRT). This information has been invaluable in helping schools and districts engage in program evaluation activities understanding, for example, how well district instruction and curriculum are

8 aligned with the state s academic standards, or how well a particular subgroup of students is performing by school, district, and across the state. Until now, however, we have been unable to answer the question, How much academic progress did a student or group of students make in one year, as measured by a criterion reference test, in relation to their academic peers? With the development of the growth model, it is now possible to answer this question. This method of examining student performance and improvement will help districts and schools to look into why results differ for certain groups of students and support discovery of which approaches are working best to help more students to higher levels of academic performance. Q: How does Nevada measure student growth? A: Nevada measures growth for an individual student by comparing the change in his or her criterion reference test achievement from one year to a subsequent year to that of all other students in the state who had similar historical CRT results (the student s academic peers ). This change in achievement is reported as a student growth percentile (abbreviated SGP) and indicates how high or low that student s growth was as compared to that of his/her academic peers. For a school or district, the growth percentiles for all students are aggregated to create a median student growth percentile for the school or district. The median student growth percentile is a representation of typical growth for students in the school or district. Q: Why didn t Nevada report growth previously? A: Nevada needed three things to be able to measure and report student growth: A statewide individual student tracking system; Statewide assessments administered in consecutive grades in the same subjects A technically sound and understandable method for measuring growth that was compatible with the assessment system. The state had all three of these components in place as of Q: When do growth reports become publicly available? A: Nevada releases public growth reports for schools and districts along with the other CRT achievement results in the fall of each school year.

9 Q: For which grades and subjects does Nevada report growth? A: Nevada reports growth for reading and mathematics for grades 4 through 8. The Nevada growth model uses students historical CRT results to calculate growth percentiles. As such, no results will be available for grade 3 (the first grade of CRT testing) or for science (because science is tested only in grades 5, 8, and high school). Q: Will all students be included in growth reports? A: No. As noted above, students in grades 4 through 8 who have two or more consecutive years of CRT results will be included in growth reporting. School- level growth reports give users the option of including students who were tested in the same school in which they were enrolled at the beginning count day of the school year, and students who were enrolled after count date. Any student who took the NAA (Nevada Alternative Assessment) at any point during the student s two most recent CRT administrations will not be included in growth reports. Q: What do growth reports show that standard CRT and AYP reports don t? A: School and district growth reports display information about how much academic progress students made in relation to their academic peers (students with a similar CRT test result history). CRT reports present information about the performance of students at the end of each school year, displaying the distribution of students performing at each of the CRT performance levels. Adequate Yearly Progress (AYP) reports display information about how close a school or district is to helping all students reach or exceed proficiency. While the measure of improvement on AYP reports is correlated with individual student growth, it is focused on comparisons of grade- level cohorts (e.g., this year s 4th graders compared to last year s 4th graders). In addition, AYP reports do not measure any change in performance for students at or above the Meeting Standards performance level. Q: Is growth a better measure of student performance than CRT or AYP? A: No. It simply answers a different question. If you want to know how well a student performed on the standards for mathematics or reading by the end of 6th grade, the CRT scaled score and performance level are the best indicators. If you want to know whether a school is on target for having all of its student s proficient by 2014, AYP reports are the right measure. If you are trying to determine how

10 much a student s performance has changed from 2010 to 2011 relative to the student s academic peers, the growth model is the best indicator. A more complete understanding of performance can be obtained by using all three measures. Q: Will growth data be used for accountability (AYP) determinations? A: Not initially. As of this printing, the Department is using growth data only as a supplement to the other CRT results we already provide. The Department may decide to submit a request to the U.S. Department of Education to use growth in accountability decisions at a later date. Q: How will growth data be disseminated to districts, schools, teachers, and the community at large? A: Growth data will be accessible to districts through a number of websites: the Nevada Department of Education website, the CCSD website & school websites will all contain growth data. Q: Will the state provide growth data for high schools? If so, how are the calculations made and how should the data be interpreted? A: Not initially. The Nevada Department of Education is currently determining when high schools will be included in the model. Q: What additional information will the Department provide to teachers, administrators, and other education stakeholders on how growth data is calculated and how to use it effectively? Where can I find this information? A: Resources including videos, links to data, this Interpretive Guide and will be located both at and on the NDE website. Questions not answered by these materials should be ed to the Growth Model help desk. Q: What is a student growth percentile? A: A student growth percentile (abbreviated SGP) measures how much a student s performance has improved from one year to the next relative to his or her academic peers: other students statewide with similar CRT test scores in prior years. The calculation answers the question, Among other students with similar CRT test score histories in previous years, what is the range of scores attained this year? The

11 model then uses the answer to determine whether a student grew at a faster or slower rate than the students peers, or at a similar rate. The statistic is interpreted as follows: if John Smith, currently a grade 5 student, has a student growth percentile of 65 in Reading, that means that John improved more between grades 4 and 5 than 65 percent of students statewide with a similar historical pattern of CRT test scores. Similarly, if John had a student growth percentile of 44 in mathematics, it means that he improved more than only 44 percent of students statewide with a similar CRT test score history. Q: How is it possible to gauge academic growth across years in Nevada when the state tests are separate instruments in each grade and there is no continuous vertical scale? A: The Nevada Growth Model is a norm- referenced growth model that quantifies the academic growth of students relative to other students taking the state assessment in a given grade and content area. The Nevada Growth Model does not quantify the magnitude of change, which would require a vertical scale, but instead focuses on the student's growth percentile which quantifies a student's progress relative to other students with similar achievement histories. Even if Nevada adopts a test in the future with a vertical scale, the student growth percentile will be the foundation for discussions of student progress. Q: Can students who perform at the top range of the Exceed Standards level show growth? A: Yes. One of the Department s criteria for selecting a model was that it had to measure growth even at the top (and bottom) of the CRT performance scale. This way, all students would have the opportunity to exhibit growth. The model accounts for this by measuring each child's growth relative to his or her academic peers. Let's suppose Jane Adams scored 480 on the grade 4 and 5 math tests. Students who scored similarly to this would comprise her comparison group she would be compared only to other students who had performed similarly on those tests. Then, in grade 6, Jane again scores 480. She would probably have a very high student growth percentile in mathematics, as most of the students in her comparison group would likely score less than 480. In fact, most students who score 480 on one test will score less than 480 on the next year s test. Only about 5% of students repeat a 480 score from one year to the next, so Jane would likely be in the 95th percentile for growth. Q: What does the median student growth percentile at my school represent? A: The median student growth percentile is the midpoint of student growth percentiles in the school. Half of the students had student growth percentiles higher than the median; half had lower. This is a

12 good way of describing the typical growth of students in the school. It is not appropriate to use the average ( mean ) when comparing percentiles. Q: Can the student growth percentile be interpreted the same way regardless of grade? A: Yes. A student with a student growth percentile of 60 improved more than 60 percent of his academic peers relative to their performance the prior year, whether that student is enrolled in grade 4, 5, 6, 7, 8. Q: Can the student growth percentile be interpreted the same way regardless of the test s subject matter? A: Yes. A student with a student growth percentile of 60 in reading improved more than 60 percent of his academic peers in reading relative to the prior year. A student with a student growth percentile of 60 in mathematics improved more than 60 percent of his academic peers in mathematics. Q: If it is the case that the median marks the place where half of our schools are above and half below each year, then won't there always be half of our schools that are characterized as doing well and half characterized as not doing well; how do we tell if things are really getting better? A: Yes, the median, by definition will always have 50% above and 50% below. When SGPs are cohort referenced, one consequence is that one is unable to see whether the system is changing over time. There are two phenomena being characterized here: the growth of students (i.e., velocity) and the change in growth of students (i.e. acceleration) over time. Both are relevant characteristics but often to different stakeholder groups. The Nevada Growth Model is focused on quantifying the former phenomena. Growth to standard approaches using student growth percentile analyses can quantify the latter. Acceleration is likely to be slow and would likely show up in other prominent education indicators (e.g., percentage of students at/above proficient). The Nevada DOE is committed to articulating the results of the growth model is a clear and truthful a manner possible and is currently investigating the possibility of creating baseline referenced student growth percentiles in addition to cohort referenced student growth percentiles. Q: Can the student growth percentile be interpreted the same way regardless of the year? A: Not necessarily. The same trajectory of scores could yield higher or lower student growth percentiles depending on the trend in performance statewide. Let s suppose a student scored 220 in mathematics in grade 4 in 2008, 222 in grade 5 in 2009, and 228 in grade 6 in 2010, and that the change from 222 to

13 228 represented a student growth percentile of 65. Now let s suppose that in 2011, the entire state s performance in mathematics improves quite rapidly, so that a substantially larger percentage of students are proficient across all grades. In that case, the same test history might represent less than 65th percentile growth, since performance is increasing overall statewide. Q: Can two students with different score histories have the same student growth percentiles? A: Yes. Each student s growth is compared only to that of others with a similar CRT test score history. The performance of Student A or Student B is compared to that of their academic peers statewide with similar trajectories. Because Student A and Student B have different score histories, Student A is not compared to Student B. Q: Can two different students with the same CRT scale score test histories have different student growth percentiles? A: Yes. This can happen for at least two reasons. First, the growth calculation takes into account a student s entire CRT tests score history, not just that of the previous year. Therefore, two students could have the same scaled scores in 2010 but different scaled scores in 2009 or earlier and therefore receive the same student growth percentile. Second, the student growth percentile metric is calculated from a transformation of the raw scores that underlie the scaled scores, not the scaled scores themselves. As many as five different raw scores can translate into the same scaled score, especially for students at the Emergent/Development and Exceeds Standards performance levels. Students with the same scaled score history may not have the same raw scores and therefore would not receive the same student growth percentiles. Q: If the median growth of my district s African- American subgroup is 59, does this mean that the average African- American student in my district grew at a faster rate than 59 percent of African- American students statewide? A: No. Students are compared to their academic peers students with similar CRT test score histories statewide not their demographic peers. The correct interpretation of a median student growth percentile of 59 for the African- American subgroup in a district is, The African- American students in my district improved more than 59 percent of their academic peers statewide. The fact that 59 is slightly higher than 50 means that the African- American subgroup performance in your district is improving slightly more than one would expect given the performance of its academic peers.

14 Q: Research shows that there are correlations between a student s demographic group and their performance on the CRT. Is the same true with growth? A: Not necessarily. The relationship between demographics and growth is complex, much more so than the relationship between demographics and achievement. For instance, because there are numerous studies that have established a correlation between economic disadvantage and achievement level, one might expect that low income students would achieve at a lower level than students without such economic disadvantages. However, it is not so clear that low income students should grow slower once you've taken performance level into account, given the way we calculate growth. Q: If my school made AYP, does that mean my students are growing faster than their academic peers? A: No. AYP determinations are based on absolute achievement and not related to growth measures. Therefore, it is possible for schools to make AYP and have low growth, if most of the school s students have achieved at the meeting standards level but their performance grows more slowly than that of their academic peers. Likewise, it is possible for a school to have most of its students growing at high rates and still not make AYP.