1 Q&A with US News & World Report The answers to the questions below (in red) come from Bob Morse, director of data research for US News, and Eric Brooks, the lead US News data research analyst for the recently released first US News Top Online Education Program Rankings. US News welcomes the opportunity to communicate with WCET s members. We hope that this will be the first of many conversations between US News and WCET about our new online degree rankings. All questions submitted by WCET members in January General Questions What is USNWR's perspective on the actual value of the first round results (apart from the obvious self-promotional brand-building aspects)? o What is the perceived value for the consumer or other audiences of a first round of rankings which so clearly does not reflect the quality of available programs? Or o Does USNWR view its findings as somehow revelatory in some way (e.g., by uncovering the four best online programs which made its honor roll)? The value of the first round rankings has more do with the fact that US News decided to do them in the first place. The timing was right because online education is no longer in its infancy. It has reached a certain level of mainstream acceptance and continues to grow. We felt that now was the time to expand our data-collection and ranking efforts into the rapidly expanding online education sector. There was also a great deal of demand for comparative information on the relative merits of online programs. Such rankings information was not available for online degree programs and US News is now starting to fill this large information gap. So, the main value in the first round is filling and information gap and taking first steps to providing evaluative comparative data on online degree programs. In the questionnaire, there seemed to be an assumption that it was desirable for distance education students to comingle with on-campus students online. Can you point to the research findings that support this concept? Answer: It s clear from our data collection that there are many ways that online education is offered and that there is not a one size fits all way to do online education. Consequently, none of the rankings gave blended programs a comparative advantage over online-only programs aside from
2 a single minor indicator about career centers used to compute the online master s in business Services and Technology ranking. But the surveys still asked many questions about the extent online programs were integrated with on-campus programs so prospective students can find the types of programs that are right for them by sorting among the characteristics of each school. How much are organizations listed paying for leads generated by your site? We are a private company and do not release that type of data, financial or web stats. What are the web analytics on the site, e.g., visits, unique visits, page visits, etc? We are a private company and do not release that type of data, financial or web stats. I ran across this quote from USNWR's marketing materials: o "And because US News & World Report is the leading ranking resource for anyone seeking an on-campus or online degree programs, you know you re getting the best advice and information available for your on-campus or online education." In what sense do the rankings they've come up with represent "the best advice and information available for your... online education"? The rankings stand by themselves. US News published a very detailed methodology that explained how programs were ranked. That methodology described each variable and weights used. Prospective students use all kinds to information to make decisions and the rankings will be one way to get comparative information about online programs that didn t exist before, in our view. There are detail profiles of programs that are part of the US News online education site. US News hopes that the rankings will only be used as one tool and not the only basis for deciding which online programs to select. How do the results of this set of rankings support their tacit assumption that rankings lead to the best available advice and information? US News is the first to say that rankings should not be used as the sole basis to choose a program and that they should be used as one tool in the
3 college/program search process. We stress the need to look at many factors in addition to rankings like cost, programs offerings, etc... Implementation Questions What specific steps does USNWR plan to take to increase numbers of participants and ensure accuracy and completeness of survey responses? We are pleased with our response rate, and expect to get a higher level of participation next year. Once schools see this year s online education rankings and the searchable individual profiles for each program, we think they may want to be part of it in the future if they can. We believe we got a very large percentage of schools with online degree programs to participate. Of course, some questions will be refined; that s just part of the process. We learned a lot doing this for the first time, and some of that will shape next year s questions. We also want to get more outcomes data, but that will take a while. Students will have to complete entire online degree programs, graduate and enter the workforce. But that s data we definitely want to get in the future. Given that USNWR has selected a set of criteria, how wedded will USNWR be to maintaining those criteria in future years to enable cross-year comparisons? Answer: U.S. NEWS is not wedded to maintaining the same methodology in order for schools or the public to do benchmarking. o Alternatively, how open is USNWR to changing those criteria to provide a more complete picture of each category? o US News has said with all academic rankings that if we are able to collect better data that enables us to improve the methodology in terms of our users we will make those changes. We have said that since this is first online degree programs rankings we expect that the methodology will evolve over time. It s not frozen or set. US News would like to have a systematic way, like an advisory group of online programs and experts in the field, to meet with us on a regular basis in order to improve what we have done. The public good to be provided in ranking education institutions, as in any ranking scheme, would be to present to stakeholders (parents, students, legislators, etc.) outcomes from an evaluation tool that can reasonably differentiate between programs based on reliable, accurate, and shared measures of quality that together sufficiently address the learning experience. Surely, this was the goal of the U.S News rankings? If so, such measures should be pre-determined and, in this case, might reasonably be
4 based on research, a detailed survey of stakeholders in the learning experience, or a serious outreach and data collection effort to the community of practice involved. Ideally, the effort to determine what ought to be measured should include all three methods. And the questions themselves should be provided to the surveyed group with enough explanatory detail that there is a reasonable assurance they are understood in the same way by respondents. o Which methods listed above did U.S News use to determine what questions to ask in the survey and why do you think the resulting set of questions is necessary and sufficient to differentiate quality between programs in meaningful ways? o In addition to examining literature reviews and academic research, US News contacted a large number of academics who run online degree programs in the areas we surveyed to do research on online education specific to that degree area and to get ideas on which questions we asked. We used those many interviews as the main basis to determine many of which questions that were asked. o What did you do to ensure the questions asked could be understood or interpreted in a consistent manner by respondents? o We did a limited amount of pre-testing the survey by sending it out to persons in each online degree program area we planned to survey to get feedback on clearness of the survey wording, etc. But a question interpreted the same by a handful of pilot testers will not always be interpreted the same across hundreds of respondents. o Can you explain how your scoring system, based on the pre-determined questions (or rubric) of quality measures, maintains its integrity as a scoring tool if you eliminate some questions because you couldn t collect the data (for whatever reason)? o We said clearly at the beginning that we were doing research from the ground up and that we had not determined the methodology in advance. The questions were designed with evaluative possibilities in mind. But as US News s published methodologies state, the 23 indicator rankings, their components, and the weights attributed to each component were determined based on quantity and quality of data collected; finalized only after survey administration was finished.
5 o How would you recommend stakeholders use the information you present here? Assuming that stakeholders mean the schools/programs US NEWS is not going to prescribe how data can be used. Some possibility is that programs can get a detailed look at specific programs by looking at their web-profiles on US News.com and if they choose can set how they stack up on the indicators US News has used in our ranking methodology. So simply put: peer to peer comparisons. How does the ranking or rating you have provided help them do this in reliable and effective ways? Answer: Based on our research this is the first time so much comparative information about the online degree areas at the program level has ever been available. US News believes that the information is reliable since it was collected in a uniform basis and that each program filled out the questions using the same definitions. All in all, the information provides other programs the most in-depth look ever at other programs. The survey as it was implemented in this iteration contained no options for community colleges to participate. Will USN be adding options for including this part of the online education community? US News is studying adding other parts of the online community for the next survey. At this point in time we don t have immediate plans of adding two year colleges. We would be interesting in finding out about the scope of online degree offering at two year colleges. In subsequent rankings, will USN provide a glossary of terms to ensure data is reported consistently across campuses? Many of the questions were defined on the survey instruments. However, we will consider doing that for the next data collection. U.S. NEWS would like to work more directly with online educators on which questions need definitions. Did USN gather input from any of the learner audiences (i.e. online or adult learners) in regard to the survey questions? US News did not seek input from learner audiences in regards to the specific survey questions or the ranking methodologies.
6 Will there be an opportunity before survey re-deployment for the questions to be revetted? Yes. US News is open to have the survey questions re-vetted. However, we would to do it an organized way. Could members of WCET assist with the vetting process? Answer: Yes. US News would like to do that in systematic way like an advisory group of WCET members to meet with us on a regular basis in order to improve what we have done and advise us. WCET members are the experts and US News would very much like to work with WCET in an organized way. Given that there was no way to provide any question interpretations, some survey questions we left blank because we didn t have the specific information in the specific way asked in the survey. It was not apparent to us how specific questions are weighted to understand the consequences of a non-response. Can you provide additional insights about issues of non-response and how that impacts on an institution s ranking? Our answer depends upon the ranking. To be included in an Admissions Selectivity ranking or a Faculty Credentials and Training ranking, schools needed to have answered all statistical questions. For example, for all programs ranked in the Admissions Selection for Master s in Engineering provided data on average undergraduate GPAs of new entrants, average GREs of new entrants, and admissions data (# of applicants and # of offers of admissions) used to calculate acceptance rates. Schools providing all these data were ranked; schools not providing all these data were not ranked (they do not appear anywhere on US News s admissions selectivity ranking table). No school was given a lower ranking for not providing statistical data for either of these indicator rankings. In contrast, the Engagement indicator rankings and Student Services and Technology indicator rankings were comprised predominantly of non-statistical questions, such as whether a business program was AACSB accredited or whether it offered career placement assistance. We believed these profile questions can be answered by all respondents so those who left them blank were scored zero points on them. Can the turnaround time for institution participation be extended in the future? Since the questions were vastly different than previous surveys, institutional staff time was tremendous and crossed multiple units of our institution. Many of the questions required a lot of research for us to complete and we really needed additional time.
7 Sorry to hear that. Indeed, this was the first year online education program degree surveys were administered to schools and understandably many were not accustomed to collecting and reporting these data at the program level, which increased the time burden. That said, the initial amount of time schools received to complete the online education surveys was comparable to what US News asks for its longstanding graduate and undergraduate surveys, while the number of questions was fewer. Schools requesting extensions were also repeatedly granted them. In fact, Master s degree programs ended up having three full months between when they first were sent the surveys and when the absolute final deadline was implemented. Bachelor s degree programs had 3.5 months. Allowing more time would not have been feasible for an annual ranking. But out of respect for survey respondents time, the 2012 survey will trim and eliminate some problematic questions from the 2011 survey. Furthermore, the upcoming 2012 online degree program surveys will once again have at least two months in the field, and probably more. Survey Questions How open is USNWR to sharing the sub-criteria behind those criteria in cases where it's not explicit? For each discipline, US News published detailed methodologies on its website outlining all the indicators used to compute each ranking. These also include descriptions on an indicator-by-indicator level of how scoring was done. Here is a sample for Master s in Education: News.com/education/onlineeducation/articles/2012/01/09/methodology-online-masters-of-education-degreerankings?page=3 o For example, what does USNWR deems as indicators for quality relative to its "maximum class size" criteria -- is it that the program states a maximum class size? Is there a particular number which is deemed "best"? If so, what are the gradations above and below that target figure? As the published methodologies explain: a school s score *for the maximum class size indicator] equals the lowest maximum class size among all [ranked] schools divided by the maximum class of [the school being scored], multiplied by the weight [assigned to
8 the indicator]. For example, if the smallest maximum class size reported by a school in the engagement and accreditation ranking was 5 students, and a school reported a maximum class size of 12 students, then its score for that indicator would be 5/12 * the number of points out of 100 that question was worth. As many totally online colleges serve adult students, have no first time students and rely primarily on adjunct faculty, will USN include factors for accurately evaluating the success of these institutions in future rankings? o How will USN develop these new measures? None of the ranking indicators had a bias against bachelor s degree programs that served adult learners or did not have first-time students. This is because only the admissions selectivity rankings incorporated student-level data, and these rankings were only produced for for master s level programs. But to evaluate the success of institutions, all of the online education degree surveys included questions on outcome measures like retention, graduation, indebtedness and career outcomes. US News may modify some of these questions for 2012 based on research and input provided by schools, and possibly, organized input from organizations such as WCET. The goal is to better incorporate these metrics into its rankings as soon as the quality and quantity of data provided by schools allow. One heavily weighted question in the student engagement category measures whether or not group work is required. Can you provide research that supports mandatory group work as a valid measure of student engagement? Yes. Student-to-student collaboration was one of the 70 quality indicators listed by 43 administrators of online education programs as part of a six round Delphi Method study in 2010 published by Kaye Shelton of Dallas Baptist University. Additionally, student interaction with other faculty and students was one of 24 quality indicators outlined in a 2000 Institute for Higher Education Policy study. Anecdotally,, US News interviewed multiple online degree program administrators who recommended student collaboration as an evaluative metric that demonstrates engagement. In the survey USN asked colleges to "guestimate" or "composite answer" broadly about all degree programs inside one college and did not look at individual degree programs.
9 There is often very little congruency across online degree programs within one academic division let alone one entire college. Do you plan to correct that? Little has been planned for 2012 so far. But US News understands there often is incongruence among programs and that this presents dilemmas for schools not wanting to report inconsistent information. Unfortunately, allowing schools to report data on more specialized levels would require specialized survey instruments with specialized questions. This would make computing rankings using standardized criteria more challenging. However, if the scope of the online education degree rankings eventually becomes expanded (as we hope), allowing schools to report on more individualized program levels may be a focus. In subsequent rankings, will you include metrics on the accessibility of the programs to students with disabilities? A couple such questions already exist. Each of the Engagement indicator rankings score the extents of programs compliance with the American Disabilities Act. Additionally, each of the Services and Technology indicator rankings score whether programs offer softwarebased readers (which vocalize print on computer screens, aiding the visually impaired). Would USN consider changing the questions regarding faculty training for instance looking at the percentage of participation in and the quality of the faculty training? This is especially important for institutions where large percentages of online courses are taught by tenured faculty, thus making it harder to mandate the training. Those are good ideas for new questions, although measuring the quality of faculty training is no simple task. Regardless, the Faculty Training and Credentials indicator ranking is not biased against tenured faculty. In fact, the rankings for each discipline placed slightly greater weight on a traditional credential (% of instructional faculty with PhDs) than toward the sums of the online training questions. The highest ranking programs were those that managed to both employ highly credentialed faculty and then train them to teach online. Ultimately, quality can be measured by what students know, do and value. Does USN plan to adapt or augment the surveys with more questions to address learner outcomes, including demonstrated outcomes that create career readiness?
10 US News is very much open to modifying the survey to include learner outcomes and demonstrated outcomes. Unfortunately, getting schools to report standardized information on either is difficult or non-existent. Programs often do not assess what their students learned in standardized ways that can be compared to other programs. Some do not track their students after graduation at all. We hope this data will become more standardized in later years. The Retention and Graduation section of the Best Online Bachelor s Degree survey included questions that seemed restrictive with regard to retention/graduation of parttime adult learners. The survey seemed to favor a first year, traditional freshman model where learners complete their degree requirements in 4 years or less. Is it possible to work with WCET and its member institutions to find alternative ways of exploring retention and graduation given the diverse learner populations we serve? Yes, input would be greatly appreciated. US News is strongly considering modifying these questions for the 2012 survey. Possible changes may include elongating the time periods students have to graduate. The bachelor s level surveys will likely also evaluate graduation rates relative to the number of course credits initially possessed by new entrants. The USNWR's rankings criteria seem skewed toward synchronous online delivery rather than asynchronous -- specifically, live tutoring, live streaming video, etc. Is this so, and if so, why? If not, what does USNWR plan to do to counteract the perception that their ranking system holds this bias toward synchronous delivery, which may be misleading to their audience? Hopefully public perception of the rankings will be rooted in understanding of the methodologies. There is no favoritism in the rankings toward synchronous vs. asynchronous classroom instruction. The Services and Technology indicator ranking scores whether live audio and live video are accessible to students, but it also equally weights whether recorded audio and recorded video are accessible to students; demonstrating no preference for one delivery method versus other. Programs that scored the best were ones with technologies that provide students the most flexibility in how they received courses. Regarding services external to course delivery, such as live tutoring or live technical support, no doubt students would prefer speaking to a person in real time for those things.
12 Comments Received Regarding the US News Survey Please feel free to write a response to any of these. We will be including them in the document we share through our blog. Of the 110 questions, 17 were completely inapplicable. And there were probably another dozen or more that were worded in such a way that I had grave concerns about answering them. I did not want to misrepresent my institution. I will try to provide examples. o First, we have three online degree completion programs: 1) the RN-BSN, 1) the Bachelor of Liberal Studies (BLS) and 3) the Bachelor of Applied Studies. I essentially answered the survey for the BLS and BAS, which are administered through the Division of Continuing Education. The College of Nursing administers the RN-BSN, and the differences in their programs and ours are significant enough that I did not think it reasonable to address them as a group. Understood. Unfortunately, US News cannot design a separate survey or create separate survey questions for each distinct online bachelor s degree program administered at institutions. o Also, there was no place to note that all students at the institution, on-campus or distance, meet the same admission and graduation criteria, and go through the same office. The only difference is that our distance education program are geared to students who have completed 60 hours of credit anywhere, though presumably at a community college. The online bachelor s degree survey contained a series of questions asking whether the program was linked to a face-to-face program, and whether each had identical standards on admissions standards, course credits, course delivery, curriculum and faculty. Maybe we can make those questions more specific next time. o Q-15: I understand the value of asking about residency. However, the institution has a residency requirement for all students, including BAS and BLS. On campus student can meet residency by taking 90 out of 120 hours on campus, 45 of their last 60 on campus, or their final 30. Because these are distance learners, the 30 hour residency rule is the ones that applies, and is modified so that distance learners must take 30 hours through the institution, not necessarily physically on campus. That was impossible to explain, given that our distance degree learners all come in with 60 hours as a requirement of admission. Also, in answering simply yes we have residency requirements, it
13 suggests an on-campus component. If I answered no it would appear as if we had lax standards. A response option for this survey question was Some of our OBD granting programs have residency requirements and some do not. This response would have most closely reflected the circumstantial nature of in-person attendance requirements at the institution. However, US News will strongly consider modifying some survey questions to account for the high prevalence of transfer students in online degree programs. o Q-18: None of the answers are true for us. Yes is misleading. No is somewhat true, insofar as the on-campus majors do operate separately from our degree programs. Not applicable is not true either though, since we do offer courses that are concurrently on campus and streamed to distance learners. So again, I have an answer, but could not provide it within the confines of the question. Not responding makes us look weak, responding within the confines is misleading. Without more specifics about the nature of the program, understanding how these responses were insufficient is difficult. The thought behind designing the response options was that online students either typically share classes with campus-based students or they do not. o Q-21: Makes no sense unless I am misunderstanding it. A student in the BAS or BLS program could certainly decide to move to campus and the credits earned would transfer they are considered institution courses. But our online and faceto-face programs are not integrated whatever is meant by that. The yes/no question referenced is as follows: Can the credits earned for courses in the online bachelor s degree programs be transferred toward the equivalent degree programs offered in the face-to-face setting? Select yes if online and face-to-face bachelor s students are integrated. The confusion involves the second part. Just because all programs with online and face-to-face bachelor s students integrated are instructed to select yes does not mean that all programs that select yes must have integrated online and face-to-face bachelor s students. Perhaps the wording needs to be modified for 2012.
14 o Q-23: Again, makes little sense. Every course we offer online is a fully vetted Institution course. The curriculum is not the same, because we do not offer majors online. The degrees offered are constructed as areas of study. US News will strongly consider modifying some survey questions to account for the high prevalence of transfer students in online degree programs. o Q-24: We don t offer majors, but by not responding it appears we just offer courses and not degrees. There was no opportunity to explain how the degrees are built, and they are indeed, rigorous degrees. o Q : Geared toward first year admission. Our students must meet Institutional admission criteria, but after 60 hours of college course work, most of the requirements for high school students are inapplicable (class rank, ACT, etc.). Some online bachelor s degree programs require applicants to submit high school level information. A majority do not, which is why this information was not used in the rankings. But no doubt prospective students benefit from being able to learn whether an individual program requires SAT scores, and if so, what those scores typically are. o Q-109: This is an example of what I consider a prejudicial question. Of course I hope our graduates are employed and employable, and they are. But are institutions required to ask this question of their on-campus graduates? As many have pointed out, the rankings would be improved if they included outcome level data on employment and salaries. Most programs do not collect this information, which is why it was not used for the rankings. o I think surveys can be valuable. And some of these questions got me thinking about how to collect this kind of data for our own purposes. However, I have studied and conducted survey research, and many of the questions here are poorly designed, and simply do not address the many ways students study online, and the many options for online degrees. US News' "honor roll" of online colleges misses the point and lacks understanding of adult learners. In more than a decade of experience, I've learned that students choose online programs that serve their needs, not institutions. However, US News bundles online programs on an institutional level, the more the better. Therefore, an institution
15 with one or two exceptional, nationally respected, online programs will be ignored. One does not have to question the validity of US News' honor roll because its weakness is all too apparent. Incorrect. Data for these surveys were solicited at the program level, not at the institutional level. US News's website clearly labels each honor roll as being at the program degree level and not at the institution level. US News is out of touch in other ways, as well. I'll offer one example. One question asks which anti-plagiarism service the university uses. The question presupposes that students will write papers. Many online programs require authentic assessment in the form of real-life projects in place of papers, but the question provides no option for twenty-first century practice. Anti-plagiarism services apply to papers, exams, or any other assignment in which students submit writing for a grade. Writing skills may play a role in all career areas. Consequently, US News believes assuming that quality academic programs administer at least some writing component at least once across their entire courses of study is fair for use as a very small ranking criterion. Also, while the use of anti-plagiarism devices was part of the Engagement ranking, the type of anti-plagiarism service the institution uses was for profile information only. Riposte Online Rankings & One Question Their arguments boil down to the following: the rankings received too little participation from schools to be useful; not enough emphasis was placed in the rankings on student level outcomes; and US News did not properly engage online education experts in developing the questionnaires. These arguments were in most cases exaggerated or misinformed. Well, no. The most problematic argument WCET makes is that we need to learn to live with rankings and the purported consumer need for them, however dubious their efficacy and utility. I suppose. Certainly the success of publications like US News and World Report confirms the use of publishing those ranking, at least for US News and their share holders. It doesn t confirm they are really used, however (see Zemsky s* research, for instance, that finds that rankings are NOT used, even among selective schools in which information about academic performance is rarely decisive. ) And the
16 successful sales of US News and World Report s rankings certainly don t confirm that rankings should be used. That success does suggest, though, that there is a pressing need to do more to educate the American population in a variety of literacies, most notably those useful for understanding the silliness of ranking complex organizations in ways similar to the ways one might rank the reliability of toasters or determining this year s sexiest man alive. That aside, as I reviewed the survey and the issue, and after one gets past the packaging, there is potentially great value in US News and World Reports initiative. As a service, a thorough analysis of the great variety of programs could be very useful. Imagine if the issue focused on creating a taxonomy of learning opportunities rather trying to cram a host of insipid assumptions about quality into a hierarchy. It is (or would be) useful to see an easily accessible presentation of the various student demographics and the different kinds of schools that are evolving to serve such great and growing diversity. It is or would be useful to do more, as WCET suggests, to understand the various selection criteria that help institutions identify their students and the associated configurations of technologies and instructional strategies they use to help their students. Such a service might be very useful contribution for helping prospective students as well as the institutions who serve them. But for the most part it is the insipidity articulated well by many others in WCET and elsewhere that is most prominent in this round of ratings. More precisely, it is, as WCET s critique suggests, the underlying assumptions about quality indicators that are so unfortunate. Elsewhere in the issue we see things like response time to identified as a quality indicator. Certainly that is one kind of expectation--common in practice and in the underlying assumption that good pedagogy depends upon a dyadic relationship between teacher and student. But there are other ways to effectively promote productive interaction. How student community and collaboration are similarly valued and together with response times formulate a linear measure of quality does confound things a bit. Similarly, the emphasis on quantity--of software packages used, technologies used, full time faculty employed or the requisite of attending training conflates quantity with quality. Maybe the biggest problem is the idea that it is the institution, not the program or discipline, which matters. And the irony is, the issue does begin to make useful descriptive program level distinctions.
17 Similarly, though WCET rightly questions the conflation of professional accreditation with engagement. It is a small concern that professional accreditation is essentially purchased by the program. Though it is not broadly known, it is for this reason that regional accreditors sometimes have problematic relationships with professional accreditation. It is not clear, in other words, that professional accreditation is always the most useful measure of quality. All that aside, it is troubling that in practice, most of us who grouse loudest about rankings are the first to post good scores on rankings on our web sites, boast about them to our donors, and polish them in hopes of attracting prospective students. Complicating all this more is maybe the most disturbing aspect of the online rankings-- the fact that they are largely self-report. If you ask me, depending upon the stakes, I m either the sexiest man alive or a very reliable toaster. So my question? How do we expand collaboration with the publishers of US News and World Report in ways that really do serve the public, students and the institutions that strive to serve them? In 2005 Robert Zemsky reported that the vast majority of students in the U.S. exercise little choice about where they attend because the primary drivers of this decision are institution location and price. Online programs notably eliminate the location issue, but, again as Zemsky reports, Even where competitive markets do exist among selective institutions in traditional higher education, information about academic performance is rarely decisive (2005, in Ewell, 2010). U.S. News definitely appreciates your observation that the rankings are controversial and that there is a big debate on their impact on students and institutions. On key point. US News very would like to do that expand collaboration in systematic way like an advisory group of WCET members to meet with us on a regular basis in order to improve what we have done and advise us. WCET members are the experts and US News would very much like to work with WCET in an organized way. US News hopes that this Q&A is the first step to setting up such an advisory group of experts in online education.
18 Given that each of the schools surveyed has diverse student populations and characteristics, it would be helpful if some of the questions allowed for further explanation or clarification. This could be in the form of an open text box where schools can provide additional information about their data. As an example, the survey question about live technical support services to students was difficult for us to answer. Often schools may not offer 24/7 live technical support, but do accept s from students at all times of the day and respond quickly to address technical issues. Unfortunately there is no way to convey this given the construction of the survey question, which could be misleading to students and the public. Accepting s from students at all times of the day is not 24/7 live technical support. It still might be very good technical support, but categorizing schools by hard distinctions is necessary for computing rankings. However, for information purposes each school s US News profile page lists whether any technical support at all was reported as being offered, in addition to 24/7 live technical support. As a non-traditional institution, our students tend to be older students who are seeking undergraduate or graduate degrees. Many of our students come from the military and only a very small percentage of them enroll directly after high school. Many of the questions focus on traditional students; that is, they ask about admissions criteria such as rigor of secondary school record, class rank, high school GPA, and standardized test scores. For schools that offer open admissions and cater to an older student population, these factors are not particularly relevant. High school level information was not used in the online bachelor s degree rankings due to the degree completion nature of most programs. However, some online bachelor s degree programs do mandate high school information from at least some applicants, which is why this information was solicited and published on their school profile pages. At the graduate level, a proxy of a quality online master s degree program is that its admissions standards resemble those one would expect for a reputable bricks-andmortar program. Older populations (on average, in their early 30s according to survey responses) are capable of taking GREs and submitting college transcripts. Retention data was requested for all students who matriculated during four most recent 12-month time frames. For schools that have open admissions and no application fee, there are many students who attempt one or two courses and then leave the university. Unlike some traditional universities, the cost to enroll and attempt courses is
19 low. Given our population of students who are typically older, working adults, and many of whom are serving in the military, a large population of our students fit the higher education definition of student swirlers. These are students that try out a number of institutions throughout their course of study to achieve a higher education degree. They are sometimes enrolled in two or more institutions, either simultaneously or consecutively. Given that these students are not limited geographically as to where they can enroll, they often shop around for what is appropriate for them at the moment. Taking into account the unique characteristics of both traditional and nontraditional schools in this respect will better inform students and the public. Yes, input would be greatly appreciated. US News is strongly considering modifying these questions for the 2012 survey. Possible changes may include, but will not definitely include, elongating the time periods students have to graduate. The bachelor s level surveys will likely also evaluate graduation rates relative to the number of course credits initially possessed by new entrants. The rankings are an important step toward providing increased transparency and accountability for schools offering online education. By focusing more on learning outcomes rather than on the qualifications of entering students, US News could provide more relevant information to students. We are encouraged that the surveys captured a great deal of information about how exams are administered and techniques are employed to ensure academic integrity. We are also encouraged by the inquiries on the types of assessment instruments that schools use to measure student learning. To follow up with this work, we would like to see a greater focus on student learning/career outcomes in future iterations of the survey and methodology. US News is very much open to modifying the survey to include learner outcomes and demonstrated outcomes. Unfortunately, getting schools to report standardized information on either is difficult or non-existent. Programs often do not assess what their students learned in standardized ways that can be compared to other programs. Some do not track their students after graduation at all. We hope this data will become more standardized in later years. GENERAL CONCERN: The survey did not provide thorough definitions of the data elements. The IR staff completing the survey needs a glossary of terms similar to what IPEDS provides to
20 ensure that data is reported consistently across campuses. (e.g. OBD student definition needs to be clarified) Below is a specific example of clarification we requested from US News. Their answers didn t really help us complete the survey. They (US News) need to define the items so that universities with our student populations can accurately answer the questions and not have to leave them blank or report estimated numbers without footnotes! Institution questions: Per our discussion, we are planning to use the following criteria as guidelines for our data collection: This Institution has 8 bachelor s degree programs that qualify as OBD. Please note that these programs include on-campus and online students and when students select a program they do not select online or face to-face. a. Enrollment Calculations: OBD Students will be defined as students in one of the 8 OBD programs who are taking at least one course online. To select cohorts for retention and graduation calculations, we will use the fall first-time baccalaureate who are taking at least one course online during their first term. b. Degree-awarded: We cannot distinguish between OBD graduates and face-to-face graduates. (For example, if we have 154 in our BBA Marketing program which is one of the 8 OBDs- we cannot distinguish who would qualify as an OBD student. Many of our students on campus take a few online courses. For Question 24, should we just list the 5 highest percentage of graduates in the 8 OBDS even though it includes face-to-face students? c. For employment statistics, we would also need to use all graduates from all OBD programs-some of whom may never have taken an online course. US NEWS response to the institution at time of survey: Advising them that A (above) is correct and that they should just leave B and C blank if they can t distinguish students. Additional response from Bob Morse: In terms of B and C, tell school we aren t going to footnote their data. If they report it as OBD, we are going to say it s OBD. We aren t going to be able have disclaimers on questions. So, if they are comfortable saying in B that the blend represents OBD, then answer. IF in C they are comfortable if the blend =OBD, then it s OK.
UNIVERSITY OF CINCINNATI: CASE STUDY OF ONLINE STUDENT SUCCESS Melody Clark Lisa Holstrom Ann M. Millacci University of Cincinnati ABSTRACT The University of Cincinnati (UC) is a premier, public, urban
Is an Executive MBA right for you? A GUIDE TO THE EXECUTIVE MBA achieve goals busy schedule support classroom contribution cost peer network the new MBA What is an Executive MBA? The Executive MBA is a
ACCEPTABILITY OF ONLINE BACHELOR'S DEGREES Journal of Computing in Higher Education Fall 2004 Vol. 16(1), 150-163. Acceptability of Online Degrees As Criteria for Admission to Graduate Programs Margaret
Programme Specification Master of Science Financial Economics Valid from: September 2014 Faculty of Business SECTION 1: GENERAL INFORMATION Awarding body: Teaching institution and location: Final award:
Technical Report No. 15 An Analysis of IDEA Student Ratings of Instruction in Traditional Versus Online Courses 2002-2008 Data Stephen L. Benton Russell Webster Amy B. Gross William H. Pallett September
Master of Healthcare Administration Frequently Asked Questions Is your program CAHME accredited? Yes, the Seton Hall MHA program received initial CAHME accreditation effective May 20, 2014. This extends
Council for Standards in Human Service Education National Standards ASSOCIATE DEGREE IN HUMAN SERVICES http://www.cshse.org 2013 (2010, 1980, 2005, 2009) I. GENERAL PROGRAM CHARACTERISTICS A. Institutional
Exit Evaluation M.Ed. Educational Leadership NO NAME PLEASE! DEMOGRAPHIC INFORMATION: Year of Program Completion: On Campus Program: Cohort Program (Off-Campus) (check which applies) Number Years in Present
Noel-Levitz Report on Undergraduate Trends in Enrollment Management 2013 Student Retention and College Completion Practices Report for Four-Year and Two-Year What s working to increase student retention
Bozkurt Identifying Stakeholder Needs 1 Identifying Stakeholder Needs within Online Education Dr. Ipek Bozkurt Assistant Professor Engineering Management Programs School of Science and Computer Engineering
Commission on Colleges Southern Association of Colleges and Schools Best Practices For Overview to the Best Practices These Best Practices are divided into five separate components, each of which addresses
Law School Rankings The U.S. News Law School Rankings: Why and How they are done. Plus comments on the current state of law school rankings Robert J. Morse, Chief Data Strategist U.S. News & World Report
The Balanced Scorecard Beyond Reports and Rankings More commonly used in the commercial sector, this approach to strategic assessment can be adapted to higher education. by Alice C. Stewart and Julie Carpenter-Hubin
NORTHERN ILLINOIS UNIVERSITY College: College of Business Department: Inter-Departmental Program: Master of Business Administration CIP Code: 52.0201 Northern Illinois University s M.B.A. program follows
June 2014 Master of Public Administration at Upper Iowa University 1 Academic or Professional Master's Degrees: Does it Matter? Yes! The UIU MPA program combines both an academic and a professional focus.
ONLINE Bachelor of Science in INFORMATION SYSTEMS ONLINE Master of Science in Management Information Systems Information Systems Online University of Alabama at Birmingham COLLAT School of Business Dear
Student Success: Progress, but Challenges The University Professional and Continuing Education Association Center for Research and Consulting In partnership with 2013 UPCEA & InsideTrack, All Rights Reserved.
Master s Degrees - A Guide to Studying Online Why Study With the University of Liverpool? As postgraduate degrees become increasingly necessary for career advancement, employees are becoming far more discerning
1 Doctor of Education - Higher Education The University of Liverpool s Doctor of Education - Higher Education (EdD) is a professional doctoral programme focused on the latest practice, research, and leadership
New Approaches to Online Accounting Education Mark Myring, Ph.D. Department of Accounting Miller College of Business Ball State University email@example.com 765-285-5100 Jennifer P. Bott, Ph.D., SPHR Department
ADMISSIONS INFORMATION PACKET Including: Application Checklist MASTERS OF HEALTH SERVICES ADMINISTRATION PROGRAM Fall 2009 Department of Health Science & Human Ecology 5500 University Parkway San Bernardino,
CHAPTER VI STANDARDS FOR NURSING EDUCATION PROGRAMS Section 1. Statement of Purpose. is: (a) The purpose of the board in adopting rules and regulations in this Chapter programs. (i) To serve as a guide
Massive Open Online Courses (MOOCs): An Assessment Based on Personal Experience By Martina Mangelsdorf, Managing Director and founder of GAIA Insights Trends come and go in education as in any other domain.
The Future of Ranking Systems The U.S. News Experience and its impact on colleges. The rise of global rankings. Accreditation versus Rankings Robert J. Morse, Director of Data Research, U.S. News firstname.lastname@example.org
ASPIRE INSPIRE LEAD RN to BSN Program 1 You can do the best work of your life! Graceland University s online RN to BSN program As an RN, you re already doing important work, and you re part of the fastest
Blended Learning vs. Traditional Classroom Settings: Assessing Effectiveness and Student Perceptions in an MBA Accounting Course Clement C. Chen, University of Michigan - Flint Keith T. Jones, Illinois
Informing Science InSITE - Where Parallels Intersect June 2002 Integration of Upper Division Business Core Classes: A Lesson in Informing Science John D. Haney and Mary Bowers Northern Arizona University,
Providence College Local Faculty Survey: Analytical Memo of Findings October 1, 2013 Overview The following memorandum provides an analytical narrative of the results of the 2013 Ithaka S+R Local Faculty
Master of Public Health (MPH) PROGRAM Area of focus: Community Health Education ADMISSIONS INFORMATION PACKET Department of Health Science & Human Ecology 5500 University Parkway San Bernardino, CA 92407
The University of Hawai i Community Colleges Online Learning Strategic Plan GOAL 1,2,3,4,5 DESCRIPTION OUTCOMES RESEARCH QUESTIONS AND DATA REPORTING FURTHER CONSIDERATIONS Overview As an important subset
North Carolina Professional Teaching Standards For every student in North Carolina, a knowledgeable, skilled compassionate teacher...a star in every classroom. As Approved by the State Board of Education
Exposure Draft Accounting Accreditation Standards (November 15, 2012) FOR AACSB MEMBER REVIEW NOT FOR PUBLIC DISTRIBUTION Eligibility Procedures and Accreditation Standards for Accounting Accreditation
Dual Credit in Indiana Q & A Version 7.8 October 30, 2012 Dual Credit in Indiana Q&A GENERAL INFORMATION 1. What is dual credit? In Indiana, dual credit is the term given to courses in which high school
Academy Administration Practice Research Project Abstracts January 2013 The following abstracts describe a sampling of projects completed by Hanover Research on behalf of various higher education institutions
THE CALIFORNIA STATE UNIVERSITY ENROLLMENT MANAGEMENT POLICY AND PRACTICES What Is Enrollment Management? It is the intent of the Legislature that each California resident with the capacity and motivation
MARSHALL UNIVERSITY HONORS COLLEGE POLICY HANDBOOK PURPOSE This handbook offers basic information about the procedures and practices of the Honors College at Marshall University. The information presented
The Journal of Interactive Online Learning Volume 1, Number 3, Winter 2003 www.ncolr.org ISSN: 1541-4914 Analysis of the Effectiveness of Online Learning in a Graduate Engineering Math Course Charles L.
USING THE ETS MAJOR FIELD TEST IN BUSINESS TO COMPARE ONLINE AND CLASSROOM STUDENT LEARNING Andrew Tiger, Southeastern Oklahoma State University, email@example.com Jimmy Speers, Southeastern Oklahoma State
Bachelor of Science in Health Systems Management School of Public Affairs Yale Gordon College of Liberal Arts University of Baltimore External Reviewer Report Conan Dickson, Ph.D., M.P.H. Administrator,
PREAMBLE Dean Kate VandenBosch charged our committee to offer recommendations related to alumni engagement over the next decade. She asked us to consider alumni interests and the needs of CALS students
The Historic Opportunity to Get College Readiness Right: The Race to the Top Fund and Postsecondary Education Passage of the American Recovery and Reinvestment Act (ARRA) and the creation of the Race to
MASTER OF EDUCATION 1 MASTER OF EDUCATION DEGREE (M.ED.) (845) 575-3028 firstname.lastname@example.org MISSION AND OBJECTIVES The Teacher Education programs offered by Marist College are integrally linked
Indiana University School of Social Work PRAC Annual Report 2005-2006 Academic Introduction The Indiana University School of Social Work was founded in 1911. It currently offers social work education at
Online Schools of Public Health and Public Health Programs * This paper addresses a specific audience: online schools of public health (SPH) and public health programs (PHP). For the purpose of this paper,
Academic Program Review SUMMARY* Department under review:_master of Business Administration Date self-study received in Dean s office: Aug. 28, 2013 Date of external consultant s review: April 24, 2013
THE DUE DILIGENCE PROCESS FOR SUB-ADVISED INVESTMENT OPTIONS Our proprietary due diligence process provides a rigorous and disciplined framework for identifying, hiring, and retaining premier investment
Dual Enrollment Dual enrollment is an acceleration mechanism that allows students to pursue an advanced curriculum relevant to their individual postsecondary interests. Each year, more than 50,000 students
SUNY College at Buffalo (BSC) Bachelor of Business Administration (BBA) Educational Effectiveness Assessment Plan Version 2010.1 BSC BBA Educational Effectiveness Assessment Plan TABLE OF CONTENTS Introduction...
THE UIS MODEL FOR ONLINE SUCCESS Bill Bloemer, Ph.D. Research Associate, Center for Online Learning, Research and Service University Of Illinois Springfield ABSTRACT This case study describes the philosophy
So you re thinking of applying to graduate school to pursue an advanced degree? It is a major decision and one that you should give careful thought and Degrees preparation. Here is a step-by-step guide
Engaging Students through Communication and Contact: Outreach Can Positively Impact Your Students and You! Wendy Achilles WAchilles@kaplan.edu Kimberly Byrd email@example.com Jaclyn Felder-Strauss JFelder-
Objective, rigorous, and useful research to understand the post-katrina school reforms. POLICY BRIEF EducationResearchAllianceNola.com August 24, 2015 SIGNIFICANT CHANGES IN THE NEW ORLEANS TEACHER WORKFORCE
Admission to Keystone College Keystone College accepts applicants on a rolling admissions basis; however, priority is given to applications received by May 1 for fall semester applications and December
AACSB (THE INTERNATIONAL ASSOCIATION FOR MANAGEMENT EDUCATION)...2 ABS (ASSOCIATION OF BUSINESS SCHOOLS)...2 ACCREDITATION...2 ACBSP (ASSOCIATION OF COLLEGIATE BUSINESS SCHOOLS AND PROGRAMS)...3 AMBA (ASSOCIATION
Preparing for Graduate School USC CAREER CENTER Is graduate school for me? Take this brief survey to assess your motivation to continue your education Types of graduate programs Learn the differences between
So You Want To Go To Econ Grad School... Tim Salmon Department of Economics Florida State University October 2006 I get asked a number of questions each year by students who are considering economics graduate
Philadelphia University SCHOOL OF Science and Health Master of Science in Midwifery Post-Baccalaureate Certificate in Nurse-Midwifery Power to Do The Midwifery Institute of Philadelphia University Midwifery
Response from: MnSCU-Lumina Graduate Minnesota Program Charting the Future of Minnesota State Colleges and Universities: Draft Report of the Strategic Workgroups 1 RECOMMENDED GUIDING PRINCIPLE YES: Make
AAP ACADEMY ADMINISTRATION PRACTICE Research Project Growth Starts With Knowledge Abstracts The following abstracts describe a sampling of projects completed by Hanover Research on behalf of various higher
Adopted: April 8, 2013 Updated: January 31, 2015 Eligibility Procedures and Accreditation Standards for Accounting Accreditation Engagement Innovation Impact AACSB International The Association to Advance
TEACHER PROGRAM FREQUENTLY ASKED QUESTIONS Admission & Program Requirements What is the first step? Review the program details and complete the online application. What are the entrance requirements? Review
INTERNATIONAL GRADUATE SCHOOL OF BUSINESS MBA IS AN MBA RIGHT FOR YOU? THE MBA THAT S S AHEAD OF OF THE THE REST REST Is an MBA right for me? Over the years we ve helped thousands answer this question.
Deploying Professionally Qualified Faculty: An Interpretation of AACSB Standards An AACSB White Paper issued by: AACSB Accreditation Coordinating Committee AACSB Accreditation Quality Committee January
Each year, Hobsons seeks to collect data from all Business Schools offering an MBA program within Australia via a survey. Once the data has been collected, it is summarised and placed on the Hobsons website
Program Proposal Master of Social Work Program (ARC Submission, Feb. 2014) Statement of Proposal: The Social Work Convening Group proposes to institute a Master of Social Work, commencing in the fall 2015
1 Social Work Program Outcomes 2009 2010 2 The 2008 Educational Policy and Accreditation Standards (EPAS) identified by the Council on Social Work Education (CSWE) include a provision for assessment of
Hello, thank you for joining me today as I discuss universal instructional design for online learning to increase inclusion of I/DD students in Associate and Bachelor degree programs. I work for the University
Six Tips for Getting Great Letters of Recommendation for College Written by: Mark A. Montgomery, Ph.D. One of the most important elements of your college application is the recommendation of a teacher.
Office of Transfer and Career Services 802-387-6823 firstname.lastname@example.org Transfer Guide: The College Search Process The College Search Process Step 1: Research Yourself... 3 Activity 1: Getting
Part I: Goals and Data Analysis Complete College Georgia Plan 2012-13 Complete College Georgia furthers the mission 1 of Bainbridge College, particularly our emphasis upon accessible and excellent education.
Communication Humor and Personality: Student s attitudes to online learning Originally published in the Academic Quarterly Exchange, Summer 2001 Diane Goldsmith Ph.D. Connecticut Distance Learning Consortium
GRADUATE SCHOOL IN RELIGIOUS STUDIES (OR RELATED HUMANITIES/SOCIAL SCIENCE FIELD) FREQUENTLY ASKED QUESTIONS Prepared by the Department of Religious Studies, University of Tennessee, Knoxville Deciding
BOARD OF REGENTS SUMMARY OF ITEM FOR ACTION, INFORMATION, OR DISCUSSION TOPIC: University of Maryland, College Park: Master of Science in Marketing Analytics (MSMA) COMMITTEE: Education Policy and Student
COLLEGE OF THE ALBEMARLE Annual Program Review Summary (Associate in General Education) (A 10 30 0) December 2004 ABSTRACT ASSOCIATE IN GENERAL EDUCATION: PROGRAM REVIEW PURPOSE Program/unit review supports
ASSESSING THE FACULTY AND STUDENTS PERCEPTIONS OF DESIRE2LEARN AND BLACKBOARD Adnan Chawdhry, California University of PA, email@example.com Karen Paullet, merican Public University System, firstname.lastname@example.org
University Department Module Overview Glasgow Caledonian Business school Business Management The Business School has been implementing assessment re-engineering using a variety of technologies, initially
2013 Y e a r i n R e v i e w www.keeplearning.uw.edu 206-897-8939 800-506-1325 Keep learning. It s the Washington Way. 2 University of Washington Educational Outreach is entering into a dynamic period.
The Reality of College Readiness 2013 Georgia Annually, ACT provides each state with The Condition of College & Career Readiness, a report that details the college readiness of students who took the ACT
Practical Nursing Frequently Asked Questions (FAQs) There are always a lot of questions about how to apply for the Practical Nursing program. Many people rely on what they hear from others because they
Understanding your ENGAGE Results 2010 by ACT, Inc. All rights reserved. 18349 What is ENGAGE College? ENGAGE College is a survey from ACT that will help you identify your strengths and needs for several
C O L L E G E P R O F I L E - O V E R V I E W Chamberlain College of Nursing Houston, TX Web Site www.chamberlain.edu/ Institution Type Private for-profit Coeducational Yes Undergraduate Students 258 Women