University of Central Oklahoma Survey on Learning Management Systems Executive Summary Prepared by the Technology Resource Center, Office of Information Technology May 12, 2011 1
Executive Summary The University of Central Oklahoma (UCO) embarked upon a campus wide project to evaluate and select a new learning management system (LMS) for the campus. The new LMS will replace UCO s current system, Blackboard Campus Edition 8 (a.k.a. WebCT, learn.uco.edu), which is at end of support life. As part of the process, the Office of Information Technology conducted a survey of students, faculty, and staff on learning management system usage, features and functionality. Results from this survey will be utilized in conjunction with other data to create specifications for the purchasing process. Respondent Demographics A total of 810 students, faculty and staff started the survey and a total of 694 completed the survey. Of those who started the survey, 611 were students and 192 were faculty. For those that completed the survey, 528 were students and 166 were faculty. UCO Learning Management System Survey: Student Respondents N=611 Graduate, 19.6% Freshman, 13.7% Sophomore, 12.1% UCO Learning Management System Survey: Faculty/Staff Respondents N=192 Part-Time Faculty, 28.1% Senior, 30.0% Junior, 24.5% Full-Time Faculty, 59.9% Administrat or, 3.1% Staff, 8.9% Additional demographic information for faculty/staff respondents showed an equal representation of respondents per college with 48.4% teaching undergraduate course levels and 37% teaching both undergraduate and graduate levels. UCO Learning Management System Survey: Faculty/Staff by Course Levels Taught N=192 N/A, 8.3% Both Undergrad and Graduate, 37.0% Graduate, 6.3% Undergrad, 48.4% UCO Learning Management System Survey: Faculty/Staff by College N=192 Other, 5.7% Math & Science, 19.8% Liberal Arts, 19.8% Fine Arts & Design, 12.5% Business Admin, 22.4% Education & Prof Studies, 19.8% 2
Learning Management System Usage Respondents were asked to describe their usage of Blackboard Campus Edition 8. Questions were designed to speak to student and faculty/staff perspectives. However, questions were organized in such a way that a comparison of student vs faculty/staff usage could be made. The primary use of the current learning management system at UCO is in face-to-face classes (77.0% faculty, 79.9% students). Face-toface 77.0% Hybrid 20.9% UCO Learning Management System Survey: Types of Courses Using Blackboard CE 8 Faculty, N=148 Face-toface 79.9% Student, N=617 Online 42.6% Research, 3.4% Other, 9.5% Hybrid 29.7% Online 45.9% Never Used 1.9% Overall, student and faculty/staff usage of features and functionality were similar. Top uses were distribution of the syllabus; submitting/posting assignments; submitting/posting documents, files and links; and grades. UCO Learning Management System Survey: Current Features and Functionality Uses of Blackboard CE 8 3
There were 32 Faculty/Staff respondents who indicated that they do not use Blackboard Campus Edition 8. Top reasons were: No need to provide in my course, system reliability, and need for training. The preferred method for training was equally distributed between instructor led, online tutorials, quick start guides and one-on-one training. An increased focus on faculty training may promote usage and an awareness of the value learning management systems for courses. Skills Assessment and Student Preparation Overall, 74.8% faculty/staff rated their level of expertise with using Blackboard Campus Edition 8 at intermediate or advanced. This may explain a decrease in the number of faculty who attend technology training sessions on Blackboard Campus Edition 8. 45.0% 40.0% 35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0% 25.3% Novice sending mail messages, posting course announcements, and posting course documents UCO Learning Management System: Facuty/Staff Rating on Level of Expertise N=170 42.4% 32.4% Intermediate - using discussion Advanced - assigning students boards, grading assignments in to groups, selectively releasing the system, and customizing content to specific students, the visual elements of the conducting real time online course site, and creating teaching using video or audio assignments and assessments tools, and incorporating external tools With a focus on helping students learn, student respondents were asked about challenges they have had in using Blackboard Campus Edition 8. Not all of my instructors use it effectively received 65.3% of the responses. Three-fourths of faculty rated their level of expertise at intermediate or higher. Lack of availability (29.4%) and challenges with individual computers (30.8%) were also notable challenges for students. 4
70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 19.8% I have not had any problems. UCO Learning Management System: Student Q. Which challenges have you encountered using Bb CE 8? N=530 65.3% 13.6% Difficult to use. 29.4% 30.8% Not available when I need it. Challenges getting it to work on my personal computer. 20.2% My instructors do not use it. Not all of my instructors use it effectively. 18.3% 16.8% Access to Technology Support for technical issues with the system. Access to my instructor for questions about the course/assignment. Student respondents were asked their opinion about the level of preparation and support they receive to use Blackboard Campus Edition 8. Of the 530 respondents, 78.6% rated the level of preparation and support at C average or above. UCO Learning Management System: Student Respondent Rating on Level of Prepartion and Support N=530 A Excellent F, 15.7% A, 16.6% D, 5.7% B Satisfactory C, 20.9% B, 41.1% C O.K. I got by D Not enough F There was preparation and support? Students were also asked about their preferred method for learning how to use a learning management system. This information will be valuable as the Office of Information Technology prepares for support and training of 17,000+ students on the new learning management system. Top preferred methods are demonstration to the class (34.4%) and online tutorials and knowledge base (31.4%). 5
UCO Learning Management System: Student Q. Which type of training would be most effective? N=529 Video tutorials, 11.9% Online tutorials and knowledgebase, 31.4% Demonstration to the class, 34.4% Printed materials, 11.9% Face-to-face training in a computer lab, 10.4% What I Want in a New LMS All respondents were asked to rate the importance of specific features and functionality that must be included in the new learning management system. Ratings on must-have features were consistent between student and faculty/staff respondents. In addition, all respondents were invited to provide additional input by completing the following open-ended statement: I wish the new LMS system would have the following features or functionality Student responses were categorized as mobile, email/notifications, functionality, performance, instructor use, user interface, training/support and other. Of these, most want access to the LMS through a mobile device, improved or required instructor use, and improved functionality. Selected statements related to these are as follows: Mobile device: I wish it was easier to access from my mobile devices. Ability to acces on ipad and iphone. Better mobile platform support. Instructor use: I wish all of my teachers were required to use it and required to post grade through it. It really helps to see your progress in the class. I just wish ALL teachers would use this. I wish that the professors were able to use the functions of WebCT in a more effective way. 6
Improved functionality: Make it so that some classes are able to cross share information. Work on ANY computer not matter where it is!! A calendar like www.google.com/calendar. File management like www.dropbox.com. Other, 9.77% Training/Support, 5.12% User Interface, 5.58% Instructor Use, 19.07% UCO Learning Management System Survey: Student, N=215 I wish the new LMS system would have the following features or functionality Performance, 9.30% NA, 3.26% Mobile, 20.00% Email/Notification, 12.09% Functionality, 15.81% Mobile Email/Notifications Functionality Performance Instructor Use User Interface Training/Support Other NA Faculty open responses were broken into two categories: wish lists and deal breakers. The top wishes for faculty respondents were improved functionality and user interface. The top deal breakers were functionality (e.g. grade book, discussion) and course conversions. Selected statements related to these categories are as follows: I wish the new system would allow us to select several items and perform a single function on all of them at the same time instead of having to select one and perform the action, then the next, and so on. Very time consuming. An easy to use grade book that does a running average. If courses currently in use that were created in Blackboard don t roll over seamlessly to the new system, that would be a deal breaker. Grading rubrics. Key findings for faculty: Courses designed in Blackboard Campus Edition 8 must convert easily and seamlessly. A robust gradebook tool is essential, yet must also be easy to use. It must also provide for online grading of rubrics. It is important to have robust feedback and interactive tools such as discussions, announcements and chat. The system must be user-friendly, flexible and reliable. Increased technology support for faculty and their students is desired. 7
Key findings for students: Students want faculty to use a learning management system and use it effectively. Anytime, anywhere computing is essential, including access to email, calendar, alerts and course notifications on a mobile device. The system must be user-friendly, flexible and reliable. Increased technology support for students is desired. Conclusion Survey responses indicate that the University community is ready to move to a new learning management system. Therefore, input from this survey and other feedback mechanisms are important to incorporate into the selection process for UCO s next learning management system. Responses also show that students, faculty and staff are fully vested in using a learning management system. It will be equally important for the University to invest in not only in the technology, but also in the extensive training and support model needed to ensure the best implementation of a learning management system in support of our mission to help students learn. 8