November Thought Leaders: The Coming U.S. News & World Report Rankings of Undergraduate Teacher Education Draft 4 17 13 U.S. News & World Report (USN&WR) will soon be putting out its first rankings of undergraduate teacher education programs in conjunction with the National Council on Teacher Quality (NCTQ). This proposal has provoked considerable reaction from both supporters and detractors. Supporters argue, among other things, that teacher education is not as strong as it could be, nor as transparent, and that such a review can help. Detractors have pointed out that most of NCTQ s previous forays into teacher education have been negative towards teacher education, they have an agenda that differs in significant ways from many teacher education programs, and they rely on limited evidence. My purpose in this brief piece is not to either support or attack these new rankings. Rather, my purpose is to provide context for better understanding whatever ranking UNI will receive when they come out. Why Are They Doing this Review? One answer, from NCTQ s Website, is NCTQ will publish the Teacher Prep Review annually to provide its main customers aspiring teachers and school districts with up to date information on where the best programs are. To me, this may be thought of as the business purpose. On the same Website they also note, With the right training, talented and motivated people can be made into great teachers. But our teacher preparation programs must be up to the task. That's why NCTQ has embarked on a review of the nation's 1,400 higher ed teacher preparation programs. Using our well honed methodology, we're working to find the programs that are doing the best job in preparing tomorrow's educators, those that need to improve and those that need to be radically restructured. Our ultimate goal? Ensuring all teachers are student ready before they enter the classroom.
This is the public policy purpose. I believe that both the business and public policy purposes are part of the reason for the rankings. What Standards Are the Review Based On? NCTQ has developed a set of 17 standards that they assert represent the potential to create quality teachers (see http://www.nctq.org/standardsdisplay.do). The list follows. THE TALENT TEACHERS NEED Standard 1: Selectivity The program selects teacher candidates of strong academic caliber. WHAT TEACHERS SHOULD KNOW Standard 2: Early Reading The program trains teacher candidates to teach reading as prescribed by the Common Core State Standards. Standard applies to: Elementary and Special Education programs. Standard 3: English Language Learners The program prepares elementary teacher candidates to teach reading to Englishlanguage learners. Standard applies to: Elementary programs. Standard 4: Struggling Readers The program prepares elementary teacher candidates to teach reading skills to students at risk of reading failure. Standard applies to: Elementary programs. Standard 5: Common Core Elementary Mathematics The program prepares teacher candidates to successfully teach to the Common Core State Standards for elementary math. Standard applies to: Elementary and Special Education programs. Standard 6: Common Core Elementary Content The program ensures that teacher candidates have the broad content preparation necessary to successfully teach to the Common Core State Standards. Standard applies to: Elementary programs. Standard 7: Common Core Middle School Content The program ensures that teacher candidates have the content preparation necessary to successfully teach to the Common Core State Standards. Standard applies to: Secondary programs.
Standard 8: Common Core High School Content The program ensures that teacher candidates have the content preparation necessary to successfully teach to the Common Core State Standards. Standard applies to: Secondary programs. Standard 9: Common Core Content for Special Education The program ensures that teacher candidates' content preparation aligns with the Common Core State Standards in the grades they are certified to teach. Standard applies to: Special Education programs. WHAT TEACHERS SHOULD BE ABLE TO DO Standard 10: Classroom Management The program trains teacher candidates to successfully manage classrooms. Standard 11: Lesson Planning The program trains teacher candidates how to plan lessons. Standard 12: Assessment and Data The program trains teacher candidates how to assess learning and use student performance data to inform instruction. Standard 13: Equity The program ensures that teacher candidates experience schools that are successful serving students who have been traditionally underserved. Standard 14: Student Teaching The program ensures that teacher candidates have a strong student teaching experience. Standard 15: Secondary Methods The program requires teacher candidates to practice instructional techniques specific to their content area. Standard applies to: Secondary programs. Standard 16: Instructional Design for Special Education The program trains candidates to design instruction for teaching students with special needs. Standard applies to: Special Education programs. OUTCOMES Standard 17: Outcomes The program and institution collect and monitor data on their graduates.
Standard 18: Evidence of Effectiveness The program's graduates have a positive impact on student learning. Needed Contextualization The contextualization I wish to supply derives from what we know about measurement and evaluation as applied to this review. First, it is important to consider on what the review is based; in this case, the standards above. It should be remembered that no single review considers everything; indeed, no single review could or should. But it is a principle that a review can only shed light on what it is looking for. At a general level, at least, I cannot argue with the value of this list of standards. It is not precisely the same list that UNI is required to use by the state and profession, but it is reasonably similar. Indeed, I think that most of the standards represent things we are already required to do. It could be pointed out that their standards are not fully validated in practice, but neither are anyone else s in the profession (see, for example, Dr. Ball s comments at www.edweek.org/media/dball_why%20we%20will%20not%20boycott%20nctq. pdf). Second, going deeper, we need to look at how each standard is operationalized; that is, what specific indicators are used and with which specific sources of evidence. It may be, for example, that two different groups might have similar standards but judge them differently. For example, I disagree with NCQ that the literature supports a conclusion that simply accepting students with higher GPAs and test scores necessarily makes for better teachers, nor is it the case at UNI that education students are at the bottom. Third, one might consider how the judgments about each individual standard are summarized to come up with a final rating and then used to rank institutions. Each of the 17 standards will be assessed by examining syllabi and other documentation provided primarily by the institution. NCTQ provides a rubric on their site for assessment, although there is only a limited amount of detail. As with
all rating systems, the devil is in the details. I am unable to determine from the NCTQ website how some of the indicators would actually be assessed, and possibly even more importantly, how the individual indicator scores would be summed to create one final number. Any ranking system has to be interpreted in terms of the comparison group. For example, if I were ranked as the worst sprinter in my group, but that group was the U.S. Olympic sprint squad, then I m still pretty fast. Likewise, if I have a classroom full of students all between 5 9 and 5 10, being the shortest or tallest of the group doesn t make much of a difference; it is difference without much importance. Fourth, are the metrics this review focuses on really those that drive good teacher preparation? The recently coined McNamara fallacy notes the problems in focusing on the easily quantifiable at the expense of what may ultimately be more important. Robert McNamara, U.S. Secretary of Defense for part of the Vietnam War, famously used reported enemy body counts as evidence we were winning the war. The historical record suggests there were other variables at work that he was not considering. In the context of ranking colleges, consider Reed College s approach. Since 1995 Reed College has refused to participate in the U.S. News and World Report "best colleges" rankings.... Reed does participate in several other well established college guides that do not assign numerical rankings to institutions, including Barron's, the Fiske Guide to Colleges, Peterson's, Colleges that Change Lives, Newsweek's College Guide, and the College Board's College Handbook. Each of these guides attempts to describe more fully the experience, student culture, and academic environment at different schools. Consistent with Reed's non participation in U.S. News rankings, the college also does not participate in Money magazine's college ranking issue. Reed College has actively questioned the methodology and usefulness of college rankings ever since the magazine's best colleges list first appeared in 1983, despite the fact that the issue ranked Reed among the top ten national liberal arts colleges. Reed's concern intensified with disclosures in 1994 by the Wall Street Journal about institutions flagrantly manipulating data in order to move up in the rankings in U.S. News and other popular college guides. This led Reed's then president Steven Koblik to inform the editors of U.S. News that he didn't find their project credible, and that the college would not be returning any of their surveys.... Reed's president, Colin Diver, cautions prospective students and parents against relying on rankings. Rankings, he says, are grounded in a "one size fits all" mentality.... (http://www.reed.edu/apply/news_and_articles/college_rankings.htm)
Fifth, we might ask if the data used to do the rankings is accurate? In the NCTQ/USN&WR review there is no independent checking. Currently in the news are reports that some universities have fudged their numbers to look better on the USN&WR rankings of colleges (see, for example, http://www.theatlanticwire.com/national/2013/02/which schools arent lyingtheir way higher us news ranking/61874/). Sixth, even given accurate data, different results can still result through a different mix of criteria and weighting. Consider hospital rankings. There are at least 15 different set of rankings, and results vary. Different data, or the same data but with different criteria and weighting, yields different results. Malcolm Gladwell gave a very thoughtful analysis of rankings in general in a recent issue of the Atlantic (http://www.newyorker.com/reporting/2011/02/14/110214fa_fact_gladwell). As an example, he used a Car and Driver comparison between a Corvette, a Lotus Evora, and a Porsche Cayman. The results were Cayman 193, Corvette 186, and Evora 182; so clearly the Cayman is best, right? Gladwell points out, however, that the comparison used Car and Driver s standard formula, the same as they would use for comparing family minivans, even though these are expensive sports cars. For example, only 4% of the final score was a subjective appraisal of how the car looks. For a minivan that might make sense, but for a sports car? Cost is also only a relatively small part of the comparison, fine for car enthusiasts, not so helpful for normal buyers. And again, the driving experience contains a substantial 36%, but for a sports car buyer should probably count for a lot more. As Gladwell points out, if one changed the weightings in different but equally, or more, valuable ways, the Lotus might win or the Corvette, based on the same factual results. His bottom line on which is really the best car, and what this tells us about college rankings: Car and Driver s ambition to grade every car in the world according to the same methodology would be fine if it limited itself to a single dimension. A heterogeneous ranking system works if it focusses just on, say, how much fun a car is to drive, or how good looking it is, or how beautifully it handles. The magazine s ambition to create a comprehensive ranking system one that considered cars along twenty one variables, each weighted according to a secret sauce cooked up by the editors would also be fine, as long as the cars being compared were truly similar. It s only when one car is thirteen thousand dollars more than another that juggling twenty one variables starts to break down, because you re faced with the impossible task of deciding how
much a difference of that degree ought to matter. A ranking can be heterogeneous, in other words, as long as it doesn t try to be too comprehensive. And it can be comprehensive as long as it doesn t try to measure things that are heterogeneous. But it s an act of real audacity when a ranking system tries to be comprehensive and heterogeneous which is the first thing to keep in mind in any consideration of U.S. News & World Report s annual Best Colleges guide. Conclusion I agree with the conclusion from Dr. Ball, Dean of Education at Michigan, as to why she is not upset at the advent of these rankings. At the University of Michigan, we have been redesigning our program, and are busy implementing and improving it. We are sure we have a lot to learn still, and hope that by studying our efforts, we will be able to keep developing and improving our approaches. We also assume that we will not do well on all the NCTQ criteria; we understand that. In some cases, we simply don t yet do certain things as well as their standards specify. In other cases, we do not seek to do things in quite the way they identify. (www.edweek.org/media/dball_why%20we%20will%20not%20boycott %20NCTQ.pdf) Thus, regardless of whether UNI is rated well or poorly, please remember that this ranking is only one potential indicator of quality, whether you are a potential student, state official, member of the general public, or other stakeholder. If you are interested in attending a teacher education program, then USN&WR s rankings could be one element of your process. But it should really be only one part of your process. You should also visit the institution, talk with students and faculty, see what they really do, talk to graduates, examine other available data, etc. For those of us at UNI, the results can be part of what we consider in thinking about our quality and areas in which we might improve. But again, it should only be a part of what we consider. We have many other sources of data, some of which are direct measures instead of merely paperwork. Likewise for others considering UNI and other teacher education programs in Iowa there is more to each of us than can be shown in this one particular ranking approach.