Multi-layered Assessment: Going Beyond the End-of-Course Surveys. Robert Zotti Director of Online Learning Stevens Institute of Technology

Transcription

1 Multi-layered Assessment: Going Beyond the End-of-Course Surveys Robert Zotti Director of Online Learning Stevens Institute of Technology Introduction This paper proposes an assessment model that consists of an analysis of various metrics that go beyond the end-of-course student survey data. Student demographics, outcomes (grades), course audits, and the use of new learning technologies all contribute to this model. To the extent possible, this model also takes into account the corresponding data for conventional courses. This multi-layered approach to assessment also contains a quality improvement path for instructors. Previous Research There are several examples of studies that look at student satisfaction in online courses. There are also several examples of how students reacted to particular practices that were used inside particular courses. These studies have provided critical insights into how individual courses can best be conducted or improved. There has even been the occasional comparison of student surveys between online courses and their face-to-face counterparts. But what is lacking from these studies is a macro-level school-wide analysis of student feedback and outcomes. In addition, there are few if any school-wide studies regarding the adoption of web conferencing and similar technologies, and how they have been received by students. Finally, all previous assessment literature related to online programs that was found during this study was focused entirely on the end-of-semester or post semester timeframe. There were no examples found of assessments that were conducted early enough in semester to allow corrective actions to be taken if needed. Similarly, there were no case studies that illustrated procedures which might need to take place in an online course if all did not go according to plan. This study looks at these dimensions at Stevens Institute of Technology, a private university that specializes in engineering, computer science, and management. Conditions and Limitations Founded in 1870, Stevens began its WebCamps online learning program in As of 2009, Stevens had delivered over 230 of its approximately 500 graduate courses online. An analysis of enrollment data (including student demographic information), survey results, outcomes (grades), course audits, and technology use provides a more complete picture that previously available of an entire university s online learning operations. The number of online courses developed specifically for undergraduates at Stevens is negligible and therefore are therefore ignored for the purposes of this study. There are a number of hybrid courses at Stevens that serve both a training and an academic role. Those students who sat for these courses for training purposes are not reflected in the school s enrollment system, and hence are not accounted for in this study. End-of-course surveys often had different sets of questions and sometimes different rating scales. In an effort to normalize this data, a composite list of questions that were represented on most surveys was used. This necessitated a large number of blank cells or missing data within the data tables. Surveys and survey questions that were specialized to a particular course were dropped. One such example of a Copyright 2009 Board of Regents of the University of Wisconsin System 1

2 survey that was not able to be normalized came from a Biotech course that asked students to rate their proficiency in conducting a specific medical procedure. No attempt was made in this study to gather follow-up data from students regarding the extent to which they were able to apply what they learned in their careers. Layer 1: Registration & Demographic Data Registration data was gathered for all graduate courses at Stevens, whether they were conducted on campus, online, at off-side locations, or in some hybrid fashion. This data goes back to the Fall 2005 semester, when the number of online courses delivered at Stevens had become large enough to become a factor in the annual planning of every major academic department. The registration data for this study goes through the Fall 2008 semester. For each course, the number of graduate and undergraduate students was noted. (Interestingly, a significant number of undergraduate students have been able to integrate online graduate courses into their study plans.) The number of students who took all of their courses online, as opposed to those who mixed online courses with the more conventional face-to-face courses, was determined. The ages and of all online students was determined and their geographic locations were plotted on regional, national, and global maps using Microsoft MapPoint software. The number of students taking graduate classes at Stevens for the first time was also noted. Finally, the instructor(s) for each course was noted. This data was used to build the most complete picture possible of the online vs. face-to-face graduate enrollment situation across the entire university. Layer 2: Outcomes (Grades) Student outcomes, or grades, were added to the enrollment tables. Student identities were masked during this data transformation, though it was still possible to determine correlations between student outcomes and survey results that were gathered later. It should be noted that the grading scale used at Stevens during the time of the study reflected the A, B, C, F scheme. Starting in the Summer 2009 semester, the grading scale was changed to A, A-, B+, B, B-. Layer 3: End-of-Course Surveys Results for over 500 online and face-to-face courses per semester were gathered and examined. As these surveys were anonymous, no analysis of graduate vs. undergraduate student satisfaction could be performed. However, overall correlations between the survey results and the student outcomes could be noted. Layer 4: Growth of Real-time Web Conferences & Recorded lectures The adoption and use of new online learning technologies presented an interesting problem in that the data was difficult to obtain, and once obtained it was often difficult to determine traceability. For example, web conferencing events and pod-casts were hosted outside the school s LMS, making it difficult to determine which classes were utilizing the technologies without extensive manual intervention. In addition, many recorded lectures could be used semester after semester, though if different teachers taught the same class, one might have opted not to use the new technology at all. Copyright 2009 Board of Regents of the University of Wisconsin System 2

3 Finally, it should be noted that the web conferencing system used at Stevens since 2004 was scheduled to be replaced in Layer 5: Course Audits Course surveys and other assessment activities are primarily reactive in nature. End-of-semester surveys are not effective at providing timely insights into problems or issues that may exist in an online class. In order to gain timely visibility into the quality of online courses and give faculty members the opportunity to implement corrective actions for current students, Stevens Institute of Technology implemented a process for conducting brief audits in the first weeks of each semester. In 2008, members of the online learning staff began using a basic audit checklist to determine the extent to which best practices are utilized in online courses. In addition, these course inspections have provided an early warning of potentially disruptive situations, such as students not logging on, or course content not being available. But to date, the biggest payback from the course audit and inspection process has had less to do with compliance issues and more to do with increasing the level of communication between faculty members and the WebCampus staff. Some of the most prominent beneficiaries of this process have been new online instructors, and instructors teaching new online courses. Layer 6: A Quality Improvement Path for Instructors As the number of online instructors at Stevens steadily climbed, the WebCampus staff turned its attention to address cases where instructors might need remedial training in online course delivery techniques, or even assistance from a more experienced faculty member. A staged approach to this issue was developed and brought before the school s curriculum committee for approval. Though never used as originally written, this procedure has provided valuable guidance in the handling of some unusual cases. Results & Discussion The data compiled during this study will no doubt provide more fodder for the now-classic debate about the relative merits of online vs. conventional education that has raged in the academic community since online courses first gained prominence in the late 1990s. More recent research by Savitz and others have suggested that the debate over which delivery mode is superior may be misguided. Instead of fixating on which delivery mode is superior, it may be more productive to simply acknowledge the different delivery modes are simply different. Perhaps more significantly, the outcomes and feedback would seem to indicate that students are generally finding what they need in online courses. The endgame of the debate may simply be that there are both good online courses, and good conventional courses. The results of this study illustrate the growth of online graduate programs in engineering, management, and computer science at Stevens Institute of Technology, a small private university, between Roughly half of all graduate classes that are offered at Stevens have both an online and a face-to-face version. There are obvious implications regarding the training of faculty, the assessment of student outcomes and feedback, and the implementation of online teaching technologies that are applicable to other institutions. Other aspects of managing the WebCampus online program at Stevens, such as the rate of growth in online programs as a whole, may or may not be transferable to public universities, community colleges, or K-12 institutions. Areas for Future Research One area of interest would be how other universities with similar academic programs are managing and assessing their online offerings. Do other schools that specialize in management and engineering degrees have a similar local, regional, and national geographic distribution of students? Are the online students Copyright 2009 Board of Regents of the University of Wisconsin System 3

4 at similar schools taking all their classes online, or are they mixing online classes with more conventional sessions? What correlations are there between age groups and satisfaction? What about between age groups and preferred technologies? Another area of interest would be the contrast in student satisfaction and student outcomes at other schools particularly between the managerial and technical disciplines. Despite all the studies that have been done to date, it would seem that the field is wide open for additional research into the overall emerging role of online learning within universities as they strive to meet the nation s 21 st century education needs. Can online programs be made timely, relevant, and engaging enough so as to be irresistible rather than simply a convenience? Are there any universities that have truly achieved this kind of online learning supremacy? Perhaps only time will tell. But perhaps that time is coming sooner than we think. References Achtemeier, S. D., Morris, L. V., & Finnegan, Catherine L. (2003). Considerations for developing evaluations of online courses. Journal of Asynchronous Learning Networks, 7(1). Arend, B. D., (2007). Course assessment practices and student learning strategies in online courses. Journal of Asynchronous Learning Networks, 11(4). Armatas, S. A. (2008). Distance learning and copyright: A guide to legal issues. Washington, DC: American Bar Association. Bali, M., & Ramadan, A. R. (2007). Using rubrics and content analysis for evaluating online discussion: A case study from an environmental course. Journal of Asynchronous Learning Network, 11(4). Dziuban, C., Moskal, P., Brophy, J., & Shea, P.. (2007). Student satisfaction with asynchronous learning. Journal of Asynchronous Learning Networks. 11(1). Lim, D. H., Morris, M. L., & Kupritz, V. W. (2007). Online vs. blended learning: Differences in instructional outcomes and learner satisfaction. Journal of Asynchronous Learning Network, 11(2). Raga, R. E., & Kleoppel, J. W. (2004). Comparison of outcomes on like exams administered to inresidence and asynchronous distance-based Parm.D. students. Journal of Asynchronous Learning Networks, 8(4.) Savitz, S.R. (2006) 12th Sloan-C Conference on asynchronous learning networks. Sener, J.. (2006). Quality matters: Inter-institutional quality improvement for online courses. Journal of Asynchronous Learning Networks, 10(1). Swan, K., Shen, J. & Hiltz, S. R. (2006). Assessment and collaboration in online learning. Journal of Asynchronous Learning Networks, 10(1). Copyright 2009 Board of Regents of the University of Wisconsin System 4

5 Author Summary Robert Zotti is the Director of Online Learning at Stevens Institute of Technology. He is responsible for managing the infrastructure that allows Stevens to deliver over 150 online graduate courses each semester. Zotti was an early proponent of adding real-time online teaching components to the school s asynchronous classes, and was given the responsibility for the deployment of web conferencing. Before joining Stevens, Zotti spent 12 years as an IT project manager, trainer, and programmer at Prudential. He hold a Masters Degree in Information Management and is currently a PhD candidate. Address: Castle Point on the Hudson Hoboken, NJ rzotti@stevens.edu URL: Phone: Fax: Copyright 2009 Board of Regents of the University of Wisconsin System 5