LMS Research Project: Final Report June 2013

Transcription

1 The University of British Columbia LMS Research Project: Final Report June 2013 Submitted by: Roselynn Verwoord, Evaluation and Research Coordinator, Centre for Teaching Learning and Technology, University of British Columbia 1

2 Contents List of Tables... 3 Executive Summary... 4 Project Description... 4 Findings... 4 Introduction and Background... 5 Report Purpose and Structure... 6 Scope... 6 Research Questions... 6 Research Methods... 7 Research Procedure... 7 Initial Online Survey... 8 Individual Interviews... 8 Follow- up Survey... 9 Participant Eligibility & Response Rate... 9 Data Analysis Findings Profile of Instructor Respondents Profile of Student Respondents Perceptions of the System Support Experience and Professional Development Communications and Engagement Research Limitations Dissemination & Engagement with Stakeholders Future Research Recommendations Acknowledgements Appendix B Instructor Survey Questions Appendix B Student Survey Questions Appendix C Interview Questions Appendix D Follow- up Survey Questions Appendix E Behavioural Research Ethics Board Application

3 List of Tables Table 1: Participant Eligibility & Response Rate, Phase One (January - April 2012), Pilot Phase... 9 Table 2: Participant Eligibility & Response Rate, Phase Two (May - August 2012)... 9 Table 3: Participant Eligibility and Response Rate, Phase Three (September - December 2012) Table 4: Instructor & Student Feelings Towards Connect at Two- Weeks and Mid- Term

4 Executive Summary Project Description From January 2012 to December 2012, the LMS research project explored the experiences of key end user groups (students and instructors) with Connect, the University of British Columbia s (UBC) new learning management system (LMS). The year- long mixed- methods study gathered qualitative and quantitative data from participants at both UBC Okanagan and UBC Vancouver through an initial online survey, follow- up survey, and individual interviews in order to identify the following: 1) the experiences of key end user groups who participated in course offerings delivered via Connect; 2) the extent to which the structures, materials, and personnel provided users with the support they needed to work within Connect; and 3) what recommendations can be made for the broader rollout of Connect across UBC. Findings The findings can be organized around three important areas related to the implementation of Connect at UBC including: perceptions of the system, support and professional development, and communications and engagement. The majority of instructor respondents did not find the new LMS user friendly or easy to learn while the majority of student respondents found the new LMS user friendly and easy to learn. However, over the course of time that instructors and students spent in Connect, their perceptions of the system became more positive. For example, both instructors and students were asked to identify how they felt during the first two weeks of their course in the new LMS and to also identify how they felt having completed half a term in the new LMS. Instructor and student responses to each statement (confident, prepared, supported, confused, overwhelmed, and frustrated) positively increased by a minimum of 2% and a maximum of 37%. When asked about different tools in Connect, instructor and student respondents reported differing levels of difficulty with the tools and different levels of tool usage. The following tools had the highest percentage of usage by instructor respondents: Grade Centre, Announcements, Assignments, E- mail, Discussion Board, and Groups. The following tools had the highest percentage of usage by student respondents: Announcements, Assignments, Grade Centre, Discussion Board, Notifications, Tests and Surveys, and Messages. The following tools had the highest percentage of instructor respondents strongly agreeing or agreeing with their ease of use: Announcements, E- mail, Contacts, Discussion Board, and Notifications. The following tools had the highest percentage of student respondents strongly agreeing or agreeing with their ease of use: Grade Centre, Assignments, Tests and Surveys, and Announcements. Overall, the majority of instructors believe they are receiving enough support and are accessing support from people primarily through instructional support staff. The majority of instructors provide some form of support to their students and teaching assistants. Many instructors also provide support to students, for each of the tools in Connect. While the majority of students report not receiving enough support, a large percentage of students also do not seek support. Those who do seek support do so primarily through talking to other students or contacting the course instructor. The majority of instructor and student respondents do not know why UBC is transitioning to the new LMS. Communicating through e- mail from instructors faculty support unit is the preferred mode of communication for information related to the LMS transition. 4

5 Introduction and Background The University of British Columbia (UBC) has a long tradition of supporting distance education and blended learning. Distance Education and blended learning provide opportunities for learners from all over the world to access learning opportunities through UBC and they provide additional ways for instructors to enhance the range of learning experiences available to students. Over the years, UBC has provided learning management systems (LMS) - - software applications that provide flexible online classroom space - - in order to support the delivery of blended and fully online course modalities. Until recently, UBC provided WebCT Vista, an LMS, to instructors and students. In 2010, UBC was informed that WebCT Vista, UBC s then current LMS, would no longer be supported as of In order to identify a new LMS for the UBC community, UBC undertook extensive testing of three LMSs, inviting the UBC community to provide feedback on possible LMSs. As a result of this process, Blackboard Learn (branded Connect at UBC) was selected as UBC s next LMS and will fully replace WebCT Vista by June To facilitate the transition from WebCT Vista to Connect, UBC developed a phased implementation process with the introduction of Connect varying by faculty. Alongside the phased implementation process, the Centre for Teaching Learning and Technology (CTLT) conducted a year- long research project from January 2012 to December The research project was focused on determining users experiences with the transition from WebCT Vista to Connect in order to inform the subsequent wider roll- out of Connect across the University, including operating procedures, educational development, and user support. To support the research project, an Evaluation and Research Coordinator (Roselynn Verwoord) was hired in January 2012 and a research team consisting of staff from CTLT was subsequently formed. The research team consisted of the following individuals: Principal Investigator: Dr. Harry Hubball, Associate Professor, Curriculum and Pedagogy, Faculty of Education; Senior Advisor, Teaching and Learning and Academic Director, pro tem, Centre for Teaching, Learning, and Technology, Phone: (604) , E- mail: harry.hubball@ubc.ca Co- Investigators: Dr. Michelle Lamberson, Managing Director and Distance Education Instructor, Centre for Teaching, Learning, and Technology, University of British Columbia, Phone: , E- mail: michelle.lamberson@ubc.ca Dr. John Egan, Senior Manager, Strategic Curriculum Services, Centre for Teaching, Learning, and Technology, University of British Columbia, Phone: john.egan@ubc.ca Roselynn Verwoord, Evaluation & Research Coordinator, Centre for Teaching, Learning, and Technology, University of British Columbia, Phone: , E- mail: roselynn.verwoord@ubc.ca 5

6 In August 2013, two additional members were added to the research team based on changes in staffing. Co- Investigators: Dr. Simon Bates, Senior Advisor, Teaching & Learning; Academic Director, Centre for Teaching Learning and Technology, University of British Columbia, Phone: , E- mail: Dr. Andrea Han, Senior Manager, Curriculum and Special Projects, Centre for Teaching, Learning, and Technology, University of British Columbia, Phone: , E- mail: Report Purpose and Structure This report presents a detailed narrative outlining all aspects of the LMS research project including the research process, data collection and analysis, and key findings. This report has three purposes: 1) to disseminate the research findings to the UBC community including those who contributed to the LMS research project; 2) to inform processes (operating procedures, professional development, and user support) at the department, Faculty, and support unit level in order to better support the broader roll- out of Connect at UBC; and 3) to contribute to other institutions understandings of LMS transitions, particularly institutions that are facing or will be facing a similar transition. Scope The LMS research project involved three phases: 1) January to April 2012; 2) May to August 2012; and 3) September to December In each phase, students and instructors involved with courses being offered in Connect, were invited to voluntarily participate in an initial online survey and an interview. Instructors were also invited to participate in a follow- up survey. Instructors who were not teaching courses in Connect and students who were not taking any courses in Connect, were excluded from the research project. The research involved participants from both UBC Vancouver and UBC Okanagan and was open to all Faculties and departments offering courses in Connect. Research Questions The research questions explored in the LMS research project included: What are the experiences of key end user groups students, staff, teaching assistants and faculty who participated in course offerings delivered via the new Blackboard Learn 9 learning management system (LMS)? To what extent did the structures, materials and personnel available provide users with the support needed to work within this new LMS environment? What recommendations can be made for the broader rollout of Blackboard Learn 9 across UBC? 6

7 Research Methods From the beginning, the research project was conceived of as a mixed- methods, multi- phase research project collecting both quantitative and qualitative data. This was to provide opportunities for participants to engage in the research in multiple ways and to provide a variety of data points (throughout the year) to gather participant perspectives on the ongoing changes to the new LMS. Online surveys were selected in order to efficiently gather data from a large number of potential participants. Individual interviews were selected in order to gather in- depth participant feedback on their individual experience in the new LMS. The interviews also provided an opportunity to explore key themes from the initial online survey and to probe deeper into some of the data with a small number of students and instructors. Research Procedure In January 2012, during the research design phase, the research team developed an initial online survey by inviting UBC stakeholder groups (Information Technology, UBC Okanagan Teaching and Learning Centre, Student Interaction Transformation Program, Centre for Teaching Learning and Technology, and others) to contribute open and closed- ended questions to the Evaluation and Research Coordinator for possible inclusion in the survey. The questions focused on several areas related to users experiences during the transition from WebCT Vista to Connect, including: communications, support and professional development, and perceptions of the system. Some of these questions were included in two versions of an initial online survey (see Appendix) that was developed: one version with questions specific to instructors experiences and one version with questions specific to students experiences. Each survey contained both open- ended and closed questions. In addition, a short open- ended qualitative follow- up survey was designed by the Evaluation and Research Coordinator and Senior Manager, Curriculum Services and intended to be completed solely by instructors in order to gather their thoughts related to their experience using Connect, at the culmination of their course. Both surveys were delivered via Vovici, an online survey tool hosted at UBC. Lastly, two separate lists of individual interview questions for students and instructors were developed by the Evaluation and Research Coordinator and Senior Manager, Curriculum Services, in order to gather detailed information related to communications, support and professional development, and system perceptions. In designing the initial online survey, follow- up survey, and individual interview questions, the intention was to invite participants to respond to the initial online survey at approximately six weeks into the term, so as to gather their perspectives on using the system for half a term. The follow- up survey was designed to be open to instructors at the culmination of their course (approximately eleven to twelve weeks into the term). The individual interview questions were designed to be asked at approximately ten to twelve weeks into their course. Prior to beginning data collection, an application to the UBC Behavioural Research Ethics Board was prepared and submitted. The application was approved in February 2012 at which point instructors, students, teaching assistants, and support staff were sent an e- mail invitation to participate in the initial online survey. Participants were given a two- week period to complete the initial online survey and received an reminder. In March 2012, participants who identified in the initial online survey that they would be willing to participate in a focus group were sent an e- mail invitation inviting them to participate in a focus group. They also received a reminder e- 7

8 mail invitation after two weeks. Given the small response rate during the pilot phase (January to April 2012), conducting focus groups was postponed until phase two (May to August 2012). In the final data collection component for the pilot phase, participants (with the exception of students who were not invited to participate in the follow- up survey) were sent an e- mail invitation inviting them to participate in an online follow- up survey. Participants were given a two- week period to complete the follow- up survey and they were sent a reminder e- mail invitation. This process (e- mail invitation and reminder e- mail invitation for initial online survey, focus group, and follow- up survey) was repeated during phase two (May to August 2012) and phase three (September to December 2012). At each stage, participants were informed that their participation was entirely voluntary and that they could withdraw from the study at any time without repercussions. In May 2012, another amendment was submitted to the BREB in order to provide willing participants with the option to participate in a focus group or individual interview (rather than only a focus group). This amendment was approved in June Initial Online Survey In January 2012, an initial online survey was developed in order to gather participants experiences with Connect. The initial online survey for instructors included 38 qualitative and quantitative questions that gathered instructors perceptions about the performance of the new LMS including ease of use and tool functionality; support (including professional development) they accessed, provided to others, and received; and overall communications about and engagement in the transition. The initial online survey for students included 31 qualitative and quantitative questions that gathered students perceptions about the new LMS including ease of use and tool functionality; support they accessed; and overall communications about and engagement in the transition. Although both surveys contained a common set of questions, there were distinct questions as well. Most of the questions were closed- ended with Likert scales or multiple- choice options and many allowed respondents to elaborate on their responses in comment boxes. The questions asked are available in Appendix A (Instructors) and Appendix B (Students) below. Individual Interviews Individual interviews were employed to collect participants experiences. Participants who completed the initial online survey and who indicated interest in participating in an individual interview were sent an e- mail invitation at the culmination of the term, inviting them to participate in an individual interview. Interviews were scheduled at a time and location that was convenient for the interviewee. Participants were presented with a consent form which they were required to sign prior to the interview. During the audio- recorded semi- structured individual interviews, participants were prompted to share one or more stories of their experience with Connect and then were asked some specific questions focusing on their perceptions of the system, support, and communication and engagement. At the culmination of the interviews, participants were invited to discuss additional topics that were not asked by the interviewer. After interviews were transcribed by an outside source, the Evaluation and Research Coordinator e- mailed all interviewees to provide them with a transcribed copy of their interview and to give them the opportunity to make any revisions, additions, or corrections. Most participants who received copies of their interview transcripts noted minor changes (spelling of names, places, etc.) that needed to be made to the interview transcript. The requested changes were made prior to analyzing the interview transcripts. 8

9 Interview questions are available in Appendix C below. Follow-up Survey A follow- up survey was employed to collect participants experiences with Connect at the culmination of their course. All eligible instructors (those teaching a course in Connect during the current term) were sent an e- mail invitation, inviting them to participate in a follow- up qualitative survey. Students were not invited to participate in the follow- up survey because the purpose of the survey was to determine specific instructor- only support services that instructors accessed during the term. Follow- up Survey questions are available in Appendix D below. Participant Eligibility & Response Rate Initially, four user groups - students enrolled in Connect courses, course instructor(s), course teaching assistant(s) as well as university staff who provided support services to these user groups during the project were eligible to participate in the study. Due to low response rates from teaching assistants and the small number of support staff, the scope of the research project narrowed in Phase Two (May August 2012) in order to focus on students and instructors. Information regarding the response rates for each phase of the project can be found in Tables 1, 2 and 3 below. Table 1: Participant Eligibility & Response Rate, Phase One (January - April 2012), Pilot Phase Invited to Initial Survey Initial Survey Submitted Yes to Interview Follow- up Survey Instructors (36%) 10 (20%) 12 (24%) *0 Students (.7%) 8 (.2%) Not invited *0 Support 17 6 (35%) 3 (17%) 7 (41%) 0 Staff Teaching Assistants 22 0 N/A 0 0 Focus Groups Completed *Note: Although ten instructors and eight students from UBC O and UBC V were willing to participate in a focus group, due to scheduling challenges we were not able to schedule a focus group during this phase of the research project. Table 2: Participant Eligibility & Response Rate, Phase Two (May - August 2012) Invited to Initial Survey Initial Survey Submitted (#, response rate) Yes to Interview (#, response rate) Follow- up Survey (#, response rate) Interviews Completed 9

10 Instructors (22%) 8 (7%) 21 (18%) *6 Students (4%) 40 (.9%) Not invited *12 Support 31 8 (25%) 4 (13%) 5 (16%) 2 Staff Teaching Assistants 32 2 (6%) 0 3 (9%) N/A *Note: Of the six individual interviews with instructors and twelve individual interviews with students that were completed during phase two, three of these interviews were conducted with instructors and students from phase one of the research project because these instructors and students did not have the opportunity to participate in an individual interview during phase one. Table 3: Participant Eligibility and Response Rate, Phase Three (September - December 2012) Invited to Initial Survey Initial Survey Submitted (#, response rate) Yes to Interview (#, response rate) Follow- up Survey (#, response rate) Instructors (17%) 90 (11%) 103 (13%) 7 Students 27,976* 895 (3%) 145 (.5%) N/A 9 Support N/A N/A N/A N/A N/A Staff Teaching Assistants N/A N/A N/A N/A N/A Interviews Completed * Of the respondents 20,553 reported UBCV as their primary campus and 7423 reported UBCO as their primary campus. Data Analysis Analysis of data from the initial online survey, follow- up survey, and individual interviews was conducted at the culmination of each term. The quantitative data was analyzed using Excel and Statistical Analysis Software (SAS). The qualitative data was coded by hand and by using Atlas.ti qualitative data analysis software. Qualitative data analysis consisted of looking for common themes and relationships. Findings Although data was collected during three phases including: Phase 1 (January to April 2012, Pilot Phase), Phase 2 (May to August 2012), and Phase 3 (September to December 2012), data from Phase 1 is not being reported on. This is primarily because the functionality in Connect changed significantly from Phase 1 to Phase Two. This was partly based on feedback from pilot participants and partly due to a significant service pack upgrade that was released and implemented at UBC towards the end of phase one. In addition, there were a very small number of eligible participants 10

11 and thus there were small response rates. The findings presented below are connected to data gathered during Phase 2 and Phase 3. Profile of Instructor Respondents Much of the data gathered during the LMS research project can be organized around three important areas related to the implementation of Connect at UBC including: perceptions of the system, support and professional development, and communications and engagement. Each of these areas will be described in detail below. The majority of instructor respondents had extensive experience both with teaching and using an LMS. For example, 76% of instructor respondents had six years or more of teaching experience at the university level (inclusive of UBC and elsewhere), followed by 18% who had two to five years of teaching experience. Seventy- seven percent of instructor respondents were using Connect to support one (or more) blended courses where the LMS supports a face- to- face class while 12% were using Connect to support one (or more) wholly online courses and 11% were using Connect for both online and blended courses. In response to how instructor respondents perceive their adoption of technology, 46% identified as innovators or early adopters of technology, 46% identified as mainstream adopters, and 8% identified as late adopters. The majority of instructor respondents (31%) identified Science as their primary Faculty, with the next largest percentage of respondents identifying Arts (26%). Twenty- five percent of respondents identified a Faculty at UBC Okanagan as their primary affiliation and 6% of respondents identified Applied Science as their primary Faculty. Response rates for other Faculties are as follows: Education (4%), Land and Food Systems (3%), Medicine (3%), Sauder (2%), Dentistry (1%), and Pharmaceutical Sciences (1%). Instructors who participated in an individual interview represented several Faculties and units from both UBC Vancouver and UBC Okanagan and included the following: new and experienced UBC instructors with varying levels of LMS experience; sessional instructors with varying levels of LMS experience; instructors who primarily teach from locations other than the Point Grey or Okanagan campuses; instructors who teach wholly online; instructors who use Connect to support a blended course; and instructors whose Faculties had fully transitioned to Connect and whose Faculties had not fully transitioned to Connect. Profile of Student Respondents The majority of student respondents reported extensive experience with an LMS. For example, of the students who responded to the initial online survey, 81% are enrolled in an undergraduate degree and 37% are in the first year of their degree followed by 23% in their second year, 20% in their third year, and 14% in their fourth year. In response to how student respondents perceive their adoption of technology, 33% identified as innovators or early adopters of technology, 57% identified as mainstream adopters, and 10% identified as late adopters. In addition, 87% of student respondents identified as a domestic student. The majority of student respondents (70%) were taking one (or more) blended courses where the LMS supports a face- to- face class at the time that they responded to the initial online survey with 22% taking one (or more) wholly online courses and 9% taking both online and blended courses. When asked how many wholly online courses students had ever taken, only 6% of student respondents had taken 6 or more wholly online courses previously, with the majority (63%) having never taken a wholly online course, 20% having taken two to five, and 10% currently enrolled in their first wholly online course. In 11

12 contrast, when asked how many blended courses they had ever taken, the majority (39%) of student respondents had extensive experience having taken six or more blended courses, 26% having never taken any, 18% having taken two to five, and 17% enrolled in their first blended course. The majority of student respondents (27%) identified Science as their primary Faculty, with the next largest percentage of respondents identifying Arts (22%). Twelve percent of respondents identified a Faculty at UBC Okanagan as their primary affiliation and 11% of respondents identified Applied Science as their primary Faculty. Education (10%), Land and Food Systems (6%), Sauder (3%), Medicine (2%), Pharmaceutical Sciences (2%), Forestry (1%), and Dentistry (1%) were also selected. Students who participated in an individual interview represented several faculties and units from both UBC Vancouver and UBC Okanagan and included the following: first year students who had not used WebCT Vista previously; mature and access students who had not used an LMS previously; upper level undergraduate and graduate students with varying levels of LMS experience; students who were enrolled in blended and wholly online courses; and students who were concurrently enrolled in courses in both WebCT Vista and Connect. Perceptions of the System Both instructor and student respondents had extensive previous LMS experience with 84% of instructors and 80% of students having previously used WebCT Visa, 24% of instructors and 7% of students having previously used WebCT Campus Edition (CE4), 24% of instructors and 55% of students having used Blackboard, and 13% of instructors and 19% of students having used Moodle. The percentage of respondents with WebCT Vista experience is not surprising given that the majority of instructor respondents had six years or more of teaching experience at the university level (likely at UBC wherein they would have used WebCT Vista) and a large percentage (47%) of student respondents were taking courses in both WebCT Vista and Connect. Some instructor respondents perceived Connect to be user friendly and easy to learn while others did not. As shown in Table 4, when asked: Overall, the new LMS is user friendly, 59% of instructor respondents strongly disagreed or disagreed with this statement and 21% strongly agreed or agreed with this statement. Instructors who participated in a Connect workshop responded more positively across all measures with one exception. Overall, this demonstrates a positive relationship between taking a workshop and perceiving the new LMS as user friendly or easy to learn. Table 4: Instructor Perceptions of Connect as User Friendly and Easy to Learn Overall, the new LMS is user friendly Strongly Agree/ Neutral Strongly Agree Disagree/Disagree All Instructors 21% 19% 59% Instructors who had taken a workshop 25% 15% 60% Overall, the new All Instructors 25% 25% 50% 12

13 LMS has been easy to learn Instructors who had taken a workshop 30% 22% 48% When asked about different tools in Connect, instructor respondents reported differing levels of difficulty with the tools and different levels of tool usage. The following tools had the highest percentage of usage by instructor respondents: Grade Centre (75%), Announcements (68%), Assignments (62%), E- mail (60%), Discussion Board (52%), and Groups (40%). The tools with the lowest percentage of usage by instructor respondents included: Peer Assessment (6%), Wikis (7%), Collaboration (Built- in chat and whiteboard) (7%), Blackboard Collaborate (9%), Tasks (9%), and Blogs (10%). When asked to rate tools in new LMS for ease of use, the following tools had the highest percentage of instructor respondents strongly agreeing or agreeing with their ease of use: Announcements (67%), E- mail (64%), Contacts (59%), Discussion Board (50%), and Notifications (46%). The tools with the lowest percentage of instructor agreement with their ease of use included: Collaboration (Built- in chat and whiteboard) (9%), Peer Assessment (10%), Groups (17%), Grading Rubrics (18%), and Tests and Surveys (18%). Slightly more student respondents found the new LMS user friendly than not with 43% of student respondents strongly agreeing or agreeing and 40% strongly disagreeing or disagreeing with the statement: Overall, the new LMS is user friendly. In response to the question: Overall, the new LMS has been easy to learn, 50% of student respondents strongly agreed or agreed with 28% strongly disagreeing or disagreeing. When asked about different tools in Connect, student respondents reported differing levels of difficulty with the tools and different levels of tool usage. The following tools had the highest percentage of usage by student respondents: Announcements (91%), Assignments (90%), Grade Centre (83%), Discussion Board (75%), Notifications (72%), Tests and Surveys (61%), and Messages (60%). The tools with the lowest percentage of usage by student respondents included: Wikis (29%), Blogs (33%), and Peer Assessment (33%). When asked to rate tools in the new LMS for ease of use, the following tools had the highest percentage of student respondents strongly agreeing or agreeing with their ease of use: Grade Centre (57%), Assignments (53%), Tests and Surveys (52%), and Announcements (51%). The tools with the lowest percentage of student agreement with their ease of use included: Blogs (22%), Collaboration (Built- in chart and whiteboard) (26%), Wikis (24%), Peer Assessment (29%), and Blackboard Collaborate (30%). During interviews, many instructor and student respondents compared Connect to WebCT Vista and many felt that Connect is an improvement over WebCT Vista. For example, Bettie 1, an off campus instructor teaching a wholly online course commented: It s better than Vista and Don, a mature first year student commented: it certainly seems to do the job. Several instructor and student respondents also spoke about the number of clicks that it took for them to access content in Connect. For example, David, a second year student commented: trying to access my test after it has been graded is a bit of a challenge. It just takes a while to click through things and 1 All named participants have pseudonyms to protect their identities. 13

14 Susan, an instructor of a blended course said: there are more things to click and more sub- menus to go into and just more stuff that you have to do, to do the same thing. Instructor and student respondents had various overall impressions of Connect ranging from finding it awkward and not intuitive to being impressed by its functionality. For example, Doug, a first year student at UBC Vancouver commented: when I first got into the website I was a bit taken aback because looking at the entire thing, it looks like it s going to be a very complicated system to use and Adam, a first year student at UBC Vancouver said: the interface of the new LMS wasn t very intuitive. It seems like it has a lot of functionality but all of that functionality just seems jumbled together. While there is so much you can do with the Connect system, it s not very well laid out. On the contrary, Carole, an instructor at UBC Vancouver said: it just seemed really user- friendly and Susan an instructor at UBC Vancouver said: It s a bit more flexible, you know. There are things I like about it. You can personalize a little bit more. Many instructor and student respondents had specific comments such as a desire to understand the organization of Connect and to be able to customize Connect. For example, Susan an instructor at UBC Vancouver said: I know this is an enormous program with hundreds of people working on it and programming it and I don t know what sort of thinking they had but it often isn t really the way that I think about setting things up. Barbara, an instructor at UBC Okanagan said: Being able to decide as some people might not want the tab structure. They might want just a long page that has different sections. We have different preferences and to my knowledge there s no way to do this in Connect. Maybe there is but then it s not obvious. In addition, Bettie, an off campus instructor said: The tabs within the application they are all grey and it s hard to figure out which ones you can use and which ones you can t use and where they are and I actually didn t even notice them and David, a second year student said: When I was doing tests online, I would just find the way the page is set up - it s bulky. In addition, Don, a first year student said: I was actually hoping to change the colour of the webpage because to me every time I access it, it s felt sort of dull and boring. Instructors and students commented about the usability of different tools and functionality in Connect including the Grade Centre, groups, online quizzes, and discussions. Their comments highlighted specific challenges with certain tools and functionality as well as the possibilities that certain functionality provides. For example, Bettie an off- campus instructor said: What was difficult was getting into the individual discussion groups. I ended up going through users and groups and once you have divided them into groups, you can go to the discussion group. It s kind of a long way around. Then you have to go to group discussion board. Barbara, an instructor at UBC Okanagan said: [G]rade centre is always a tricky one and I find it a little confusing. All these buttons up on top - discover content, work offline, reports - never used reports. I m not sure what that is. I still download and do it in Excel because I find it s confusing and I guess I don t trust it 100% that the calculation in here would be 100% accurate and then I m always uncertain what the students actually see. Speaking about the relationship between functionality and pedagogy, Susan, an instructor at UBC Vancouver said: [Q]uestion sets which are really pedagogically useful in a large class - Connect is such that it won t give you the data on those questions which is very frustrating. What I ended up doing 14

15 was I just didn t look at that data which then is a pedagogical barrier. I don t know how my students did. It kind of influences my pedagogical choices which is limiting. Some respondents appreciated the functionality including Don, a first year student at UBC Vancouver who said: My professors are always making announcements on that particular thing and I will see it. It s pretty useful because sometimes it reminds me, oh, you have an assignment due. Several instructor and student respondents commented about not always understanding the titles or language used to name or describe functionality in Connect. For example, Barbara, an instructor at UBC Okanagan said: I don t have the solution myself for how to make it more intuitive so that you actually know what s under those categories like course tools, evaluation, grade centre. If I am dealing with grades do I go into evaluation? Do I go into grade centre? I don t know. Don, a mature first year student at UBC Vancouver also commented about the language in Connect saying: I note there were a couple of instances that I thought was really good, clear, simple English. And there w[ere] some other messages where it was quite hard to decode even for a native speaker what was actually being said. Susan, an instructor at UBC Vancouver said: what it actually does isn t the way that I would interpret the language in speaking about the language used in Connect. In the follow- up survey, which was distributed at the end of each term, when asked about their impressions of Connect instructor respondents had conflicting impressions on aspects of the system including: user friendliness; speed; functionality; and system organization. For example, many instructor respondents commented that the system was unintuitive while others reported the opposite. Examples of comments about the system being unintuitive included: I am still feeling like a novice using this system. I don t find it very user friendly. It worked well for what I needed it for, although it was not intuitively obvious to use. Overall, Connect is a good system but not an intuitive one. Examples of comments about the system being intuitive included: Fairly user- friendly once you get the hang of it. It s relatively easy to use once you get into it. Instructor respondents also perceived the system as slow. Examples of comments about the system being slow included: I found it slow and awkward, so I tried to keep my activity to a minimum. It is very slow. Much slower than WebCT. Would be better if response was sped up. Often find the system response slow when using off- campus. In particular, there can be significant latency in character echoing when composing text. In the follow- up survey, instructors also commented on the functionality of the system with some instructors perceiving that the system provides additional and better functionality than WebCT Vista with others perceiving that the system does not provide as much functionality. Examples of comments about the system providing increased functionality include: I m pleased with the improvement in many aspects of the system compared to what we used before. I like the features in general. Quite useful to use once we are familiar with the features. Favourable, easy to navigate, useful HTML window in grade centre. Examples of comments about the system not providing as much functionality as WebCT Vista include: 15

16 Arguably less performant than Vista. In many aspects important to me, it lacks the capabilities of Vista. I did not find the benefit of having moved to this app from Vista and in fact, for me there were sizeable disadvantages to its use. Specific tools that instructors commented on included the Grade Centre, Announcements, Groups, and Notifications. Instructors comments about the Grade Centre were primarily focused on the challenges that they have had with this tool. For example, some comments included: The grade centre is hard to use. Very clunky gradebook. I need to learn how to do grades better because it introduced numerous problems. Comments about other tools included: I m pleased with some of the tools used to create and manage groups. It has some very powerful functions such as the group tools. I like the announcements option. In sharing their impressions of Connect, instructors also commented on the organization of Connect including the language that is used, the possibility for customization of the system, and the aesthetic elements of Connect. Some examples of instructor comments about how Connect is organized included: The page/content collection structure is confusing; Some items are a little hidden and take a long time to find; and There are subsections in it that could be split into sections. One example of a comment about the language used in Connect includes: The terminology employed by its user interface is not transparent. Instructors also mentioned their desire to customize the system. For example: There should be a way to streamline [Connect s] appearance and functionality. An example of a comment about the aesthetics of Connect includes: The displays on the course content page [are] colourful. Some problems with the font disappearing in certain styles though. Support Experience and Professional Development Instructors were asked to identify how they felt during the first two weeks of their course in the new LMS and to also identify how they felt having completed half a term in the new LMS. At two weeks, 32% of instructor respondents and 36% of student respondents strongly agreed or agreed that they felt confident. This number increased to 55% for instructor respondents and 67% for student respondents at half way through the term, representing a positive increase of 23% and 31% respectively. Instructors who attended a Connect workshop responded more positively across all measures. The percentage of instructor respondents who felt prepared, supported, confused, overwhelmed, and frustrated at two weeks into the term and half- way through the term and the impact of having taken a Connect workshop or not having taken a Connect workshop is displayed in Table 4. Advanced statistical analysis of the student and instructor data revealed some important findings regarding students and instructors perceptions of support and finding the system easy to learn. For example, students who felt supported (strongly agreed or agreed with the statement: Now, having completed half a term/course in the new LMS I feel supported ) at half- way through the term, are nine times more likely than students who did not feel supported (strongly disagreed or disagreed with the statement) to give a favourable response to the question Overall, the new LMS is easy to learn. Similarly, instructors who felt supported (strongly agreed or agreed with the statement: Now, having completed half a term/course in the new LMS I feel supported ) at half- 16

17 way through the term are eleven times more likely than instructors who did not feel supported (strongly disagreed or disagreed with the statement) to give a favourable response to the question Overall, the new LMS is easy to learn. These findings indicate that for students and instructors, there is a relationship between feeling supported and finding the system easy to learn. It also highlights the significance of ensuring that students and instructors feel supported in order to increase the likelihood that they will find the system easy to learn. In addition, upon examining student responses by year of studies, we found that students in their first year of studies are two times more likely to find the new LMS easy to learn compared to those in their fourth year of studies, provided that the level of support is held constant. There are no significant differences in the odds ratios of second and third year students when compared to fourth year students. Table 5: Instructor & Student Feelings Towards Connect at Two- Weeks and Mid- Term Strongly Agree/ Agree Neutral Strongly Disagree/ Disagree All Instructors Two weeks: Prepared 32% 25% 43% Mid- term: Prepared 53% 26% 21% All Students Two weeks: Prepared 33% 27% 40% Mid- term: Prepared 62% 26% 12% Instructors who took a Connect Workshop Instructors who did not take a Connect workshop Two weeks: Prepared 36% 22% 42% Mid- term: Prepared 57% 26% 17% Two weeks: Prepared 27% 30% 44% Mid- term: Prepared 49% 27% 24% All Instructors Two weeks: 60% 20% 21% Supported Mid- term: Supported 62% 22% 16% All Students Two weeks: Supported 32% 34% 34% Mid- term: Supported 45% 34% 21% Instructors who took a Connect Workshop Instructors who did not take a Connect workshop Two weeks: Supported 62% 17% 21% Mid- term: Supported 63% 23% 14% Two weeks: Supported 56% 23% 21% Mid- term: Supported 60% 21% 19% 17

18 Strongly Agree/ Agree Neutral Strongly Disagree/ Disagree All Instructors Two weeks: 55% 19% 27% *Confused Mid- term: *Confused 25% 27% 48% All Students Two weeks: *Confused 59% 16% 25% Mid- term: *Confused 22% 25% 54% Instructors who took a Connect Workshop Two weeks: *Confused 52% 24% 25% Mid- term: *Confused 27% 29% 44% Instructors who did not take a Connect workshop All Instructors All Students Instructors who took a Connect Workshop Instructors who did not take a Connect workshop Two weeks: *Confused 58% 13% 30% Mid- term: *Confused 21% 25% 54% Two weeks: 41% 20% 39% *Overwhelmed Mid- term: *Overwhelmed 21% 23% 56% Two weeks: 36% 24% 39% *Overwhelmed Mid- term: *Overwhelmed 16% 25% 59% Two weeks: 39% 27% 34% *Overwhelmed Mid- term: *Overwhelmed 23% 20% 57% Two weeks: 44% 11% 45% *Overwhelmed Mid- term: *Overwhelmed 20% 25% 55% All Instructors Two weeks: 58% 15% 27% *Frustrated Mid- term: 44% 19% 37% *Frustrated All Students Two weeks: Frustrated 48% 18% 34% Mid- term: Frustrated 34% 20% 46% Instructors who took a Connect Workshop Two weeks: *Frustrated 54% 18% 27% Mid- term: *Frustrated 46% 19% 34% Instructors who did not Two weeks: *Frustrated 62% 12% 26% take a Connect workshop Mid- term: *Frustrated 40% 19% 41% *Note: This is a negative polarity, which means that disagreement is positive. 18

19 The majority of instructor respondents attended a new LMS workshop prior to the start of their course in Connect and felt that the support structures put in place have helped them to be successful with the new LMS. For example, 54% of instructor respondents attended a new LMS workshop at UBC prior to the start of their course in Connect and 54% of instructors attended a new LMS workshop at some point during the term that their course was offered in Connect. Forty- eight percent of instructor respondents strongly agreed or agreed with the statement: The support structures put in place have helped me to be successful with the new LMS and only 29% of instructor respondents strongly disagreed or disagreed with this statement. These numbers increased to 52% (a positive increase of 4% when compared to instructors who had not taken a workshop) and 28% (a positive decrease of 1% when compared to instructors who had not taken a workshop) among instructors who had taken a new LMS workshop either prior to the start of their course in Connect or during the term that their course in Connect was offered. Overall, this demonstrates a positive relationship between taking a workshop and perceiving that the support structures put in place have helped instructors to be successful with the new LMS. In addition, 48% of instructor respondents strongly agreed or agreed with the statement: Overall, I have received enough technical support to effectively teach in the new LMS. When asked about support that instructors provided to their teaching assistants (TAs), 60% of instructor respondents identified that they provided support to their TAs. Similarly, when asked to identify if they provided support to either their teaching assistants or students to help them use different tools in the new LMS, instructors identified providing support for the following tools: Grade Centre (43%), Assignments (41%), Discussion Board (31%), E- mail (29%), Groups (26%), and Tests and Surveys (19%). Instructors also identified people who provided support with 78% of instructor respondents selecting instructional support staff as providing support. In addition, 47% of instructor respondents identified attending a new LMS workshop and 47% identified talking to other colleagues as sources of people support that they accessed. Less frequently used sources of people support included: contacting the course teaching assistant (10%), attending a community of practice meeting (15%), and contacting the UBC IT Help Desk (24%). When asked to identify resources that instructors sought in order to use the new LMS, accessing resources on the lms.ubc.ca website was the most frequently selected resource with 58% of instructor respondents selecting this option. In addition, 48% of instructor respondents identified visiting the new LMS Help Website and 34% identified viewing manuals as resources that they accessed. Less frequently used resources included: viewing screen casts (23%) and accessing various resources on the internet (24%). A large percentage of student respondents did not feel that the support structures put in place helped them to be successful with the new LMS. For example, 36% of student respondents strongly disagreed or disagreed with the statement: The support structures put in place have helped me to be successful with the new LMS and only 24% of student respondents strongly agreed or agreed with this statement. The majority (40%) of student respondents were neutral about whether the support structures put in place have helped them to be successful with the new LMS. When asked to identify where they sought support to use the new LMS, talking to other students was the most frequently selected source of support with 38% of student respondents selecting this option. In addition, thirty- five percent of student respondents identified that they have not sought support and 34% of student respondents identified talking to the course 19