Perceptions on the educational value of a Learning Management System in a residential South African University

Transcription

1 Perceptions on the educational value of a Learning Management System in a residential South African University Scheepers, MD and Delport, R University of Pretoria Key words: blended learning, learning management system, educational value, level of use Abstract: The University of Pretoria has been using a Learning Management System (LMS) since 1998 to create a blended learning environment. The level of use and factors associated with use were investigated employing a questionnaire that was developed from guided focus group interviews with lecturers. Results indicated that62% of faculty regularly use the LMS, and that they ascribed the value added to their teaching mainly to the fact that the LMS provides an ubiquitous repository for learning resources, that it provides functionalities that alleviate the administrative burden of teaching and assessment and that students can engage with important aspects of the content through online discussions. The focus of current training strategies of the Department of Education Innovation should change to address the needs of academic staff, enabling them to move to higher levels of use and in doing so, impacting positively on student learning. 1 Introduction The University of Pretoria (UP), a residential university in South Africa with more than students, follows a blended learning approach ranging from purely face-to-face to purely internet-based depending on the needs of each particular module and its participants. The UP has been using various WebCT versions as their Learning Management System (LMS) since The strategic importance given to the institutional use of the LMS, called clickup within the institution, is mirrored by the fact that the executive of the UP includes the use of the LMS in strategic plans and performance indicators of the university. The Department for Education Innovation (EI) is responsible for staff development and endeavours to ensure optimal and appropriate use of the learning platform as part of a drive to transform teaching and learning. After a decade of dedicated effort by EI, training and supporting lecturers in the use of the system, an audit of the 2007 clickup modules revealed that 80% of these modules used only the most basic functions within the system. Only 6% of modules used the functions within the system to its full advantage to facilitate and manage learning. These results revealed that the University of Pretoria has not yet made cost-effective use of the considerable resources invested in clickup. 1 ICL 2010 Proceedings Page 1065

2 Coates, James and Baldwin predict that the use of a LMS in universities will continue to deeply affect teaching and learning in higher education [1]. They warn that the use of an LMS should encourage effective pedagogical practices and should go beyond the mere transition of text. According to Hall [2], the Achilles heel for technology adoption in education is a lack of understanding how to assist teachers to fully implement and integrate the use of technology in their teaching and learning practice. This adoption takes place where technological innovation intersects with the dynamics of personal and organisational change [2], and is therefore complex, inherently social, [and] developmental [3]. Change, in this case the adoption of a technological innovation, is not an event, but a process of implementation that is dependent on skills of teachers to use technology, infrastructure and bandwidth, and the training teachers receive. As was fully appreciated by the UP, users of such technologies develop through a gradual process of trial and error as they learn to progressively use the technology in a more confident and expert manner to address the learning outcomes [2]. There are many factors that influence the success of a blended learning approach. Birch and Burnett [4] created a framework that identifies institutional, individual and pedagogical factors as the main factors impacting on academics in their decision on how to integrate educational technology in their teaching practice. This framework shows that individual and institutional history with regard to e-learning will impact on the future direction of e-learning. Furthermore, Samarawickrema and Stacy [5] concluded from their study that the motivation of staff and their approaches to change and learning has a bigger impact on technology adoption than on their technical skills and their preference to use technology. They identified a number of attitudes among academics that impede the adoption of an LMS, inter alia the belief that online learning is inferior to face-to-face approaches and that it did not add value to learning. The aim of our study is to determine what factors contribute to the level of adoption of the LMS within the unique UP environment The approach in the study addresses similar issues as suggested in a recent review by Hall [2] in which he proposes that four questions be asked during the dissemination of an innovation: 1. Is it being used? 2. How well is it being used? 3. What factors are affecting its use/nonuse? 4. What are the outcomes? Therefore the objectives of this study are to: Assess general perceptions with regard to new technologies; Determine current use of clickup; Evaluate perceptions of faculty with regard to the educational value added by an LMS; Identify the functions most commonly used within clickup, as well as reasons why additional features of clickup are not explored; and Develop an appropriate institutional training and support provision plan with the ultimate objective of transforming the current teaching and learning model. 2 ICL 2010 Proceedings Page 1066

3 2 Methodology 2.1 Research design The design of the study was cross sectional and a mixed method design which employed both qualitative and quantitative methods was used. (Only the quantitative findings are reported here). The study was conducted in two phases: firstly focus group interviews were held with lecturers using a semi-structured interview guide to determine what the joys and concerns of faculty members were regarding the use of the LMS. This phase informed the design of the next phase which entailed the development of a questionnaire that was used to acquire quantitative data from lecturers to determine the extent of the different concerns. Stratified random sampling was used in the first phase for the focus groups. A random sample of six to ten lecturers who do use clickup and another six to ten lecturers who do not make use of clickup were selected from each group of faculties. The faculties were grouped together for the focus group interviews as follows: Group 1: Veterinary and Health Sciences; Group 2: EBIT & Natural and Agricultural Sciences; Group 3: Humanities; Group 4: Economic and Management Sciences; and Group 5: Theology, Law & Education. Focus group discussions were conducted separately with users and non-users of clickup within each faculty group. To prevent bias due to the presence of an EI staff member within the focus groups external researchers were employed to fulfil the function of both moderator and observer. The moderator used a semi-structured interview schedule to facilitate the interviews while the observer observed the process and took notes. An audio recording was made of each interview. Transcripts were made of the audio tapes. The data of the focus groups was gathered and analysed using content analysis. Selective reduction was used to determine relevant themes embedded within the data. The data was read, coded and categorized by drawing out words, phrases and segments. These categories were then analyzed in an attempt to find any patterns emerging from the data. The categories that emerged from the data also constituted the categories in which the items within the questionnaire were arranged, while the topics described the items within each category. The questionnaire had items that had a closed form, e.g. checklist items, items with 5-point Likert scale, as well as ranked items. Some open-ended questions were also posed to the participants to allow them to report other opinions not captured within the closed questions. The questionnaire was pre-tested by a few lecturers to ensure that the items in the questionnaire were explicit. Convenience sampling was used for the questionnaire as it was distributed via an online system to all lecturing staff (1225 lecturers who use click UP, and 254 lecturers not using clickup). Participation was anonymous and voluntary. A total of 432 lecturers completed the questionnaire, which provided a 29% response rate of the entire teaching population of the University of Pretoria. 3 ICL 2010 Proceedings Page 1067

4 2.2 Data analysis Data analysis of the closed questions within the questionnaire included descriptive statistics, basic frequencies, cross-tabulations and chi-square analysis to compare and explore relationships. Statistical analyses were performed with Statistix 9, Analytical Software, Tallahassee, FL, USA. 3 Results The distribution of the 432 respondents amongst the nine faculties: Natural and Agricultural Sciences, Economic and Management Sciences, Education, Law, Theology, Humanities, Engineering Built Environment and Information Technology, Health Sciences and Veterinary Science were 18, 16, 7, 3, 2, 16, 21, 11 and 9% respectively. Eleven percent of the sample were junior lecturers, 33% lecturers, 29% senior lecturers, 12% associate professors, nine percent professors and seven percent were heads of departments. The general perceptions with regard to new technologies are depicted in Figure 1. The typical use of clickup as reported by respondents is illustrated in Figure 2. Figure 1 Preferences with regards to new technologies Percentage I am skeptical of new technologies and use them only when I have to I am usually one of the last people I know to use new technologies I usually use new technologies when most people I know do I like new technologies and use them before most people I know I love new technologies and am among the first to experiment with and use them Figure 2 Use of ClickUp Percentage Daily for most teaching activities Regularly for many teaching activities Seldomly Past but do not current use Never used clickup 4 ICL 2010 Proceedings Page 1068

5 Users were motivated to use clickup because of the educational benefits for their subject/module (61% of respondents) and because it saves their time (54% of respondents). The value added to their teaching (Table 1) was mainly ascribed to the fact that the LMS provides a repository for learning resources that students can access, that it provide functions that assist lecturers with the administration and that students can engagewith important aspects of the content through online discussions. Table 1 Reported added value afforded to teaching by clickup It is a repository for learning resources that students can access It enables me to enhance students engagement with important aspects of the content through online discussions It allows students to test themselves on the content through self tests or quizzes Not at all 4.47% (8) 28.69% (101) 34.92% (125) To some extent 31.28% (56) 32.67% (115) 26.54% (95) To a large extent 36.87% (66) 14.49% (51) 11.73% (42) Not applicable 27.37% (49) 24.15% (85) 26.82% (96) It enhances academic writing by monitoring plagiarism 32.43% (119) It assists me with the administration of my module/s 14.29% (35) 23.43% (86) 32.24% (79) 14.99% (55) 28.57% (70) 29.16% (107) 24.90% (61) Table 2 provides information on the functions that were employed by the responders in the LMS. Numbers of users are expressed as a percentage of the total number of respondents, followed by the actual number in brackets. Table 2 Functions performed within clickup by responders Use of the Roster tool to display student profiles 69.1% (248) Communication via the Chat tool 65.7% (245) Supplying rubrics/grading sheets through the Grading forms 60.4% (212) Uploading marks onto the Gradebook 60.2% (195) Surveying students in my module/s 56.3% (196) Managing student registration for Computer Based Tests (CBT s), other tests or practicals 54.1% (184) (Group Manager) Communication via the tool 53.7% (188) Provision of library pages with links to additional learning resources 53.3% (176) Communication via the discussion blog & reflection tool for educational purposes 52.3% (183) Monitoring plagiarism (Turnitin assignments) 49.1% (170) Tracking student activities 49.0% (167) Using Quizzes for self assessment 49.0% (163) Providing important dates via the Calendar tool 48.7% (153) Submission of assignment 48.5% (150) Selectively releasing content/assignments/quizzes to students depending on certain criteria 46.9% (145) Providing multimedia materials (photos pictures sound animations) 46.5% (148) Uploading marks in PDF format 45.9% (111) Posting external links (outside web sites) on clickup 37.7% (109) Assisting with the management of large classes 32.6% (93) Sending announcements to students 24.1% (38) Distributing class notes to students 21.5% (32) 5 ICL 2010 Proceedings Page 1069

6 Providing administrative information to students 18.2% (28) Providing access to study guides 15.5% (19) The association between use- and non-use of clickup and agreement with reasons for not using additional features of the LMS is reported in Table 3. The user group reportedly uses clickup either daily for most teaching activities (14%), regularly for many teaching activities (47%), or seldomly (20%), and non-users used clickup in the past but not current (1%), or never used clickup (16%). Data for subjects that regarded the specific reason as not applicable were not included in the analysis. A statistically significant association was observed between the problems and limitations relating to clickup and clickup use, in that more non-users agreed with the statement than users. Effective use of other software / technologies to support teaching and learning was significantly associated with use, in that more non-users agreed with the statement than users. Large classes did not appear to discourage use of more LMS features, but the number of users that agreed with the statement did appear to be slightly, though not statistically significantly higher than but the number of non-users that agreed with the statement. Table 3 Reasons why additional features of clickup are not explored Reason Agree: % users Agree: % non-users Chisquare value p-value The problems/limitations relating to clickup prohibit 38.6% 70.0% me to explore and make use of more features A lack of awareness and exposure to more features 71.9% 82.6% Personal time constraints 71.2% 75.0% I effectively use other software / technologies to 57.1% 78.3% support my teaching and learning Large classes prohibit me from using more features 32.5% 11.1% Concerning training and support for use of clickup in teaching (Table 4), it was evident that past clients attended Face-to-face workshops (46%), or were assisted by other academic staff members (32%) or by e-learning support staff (29%). Prospective clients mainly prefer customized sessions based on the needs of their own academic department (58%), fact sheets/ step-by-step sheets with instructions (in words) on where to click (53%), concise training sessions similar to video conferencing that they could attend on their own computer in their own office (53%), online tutorials (51%), and short video clips demonstrating how to use a tool (49%). Respondents who never used clickup and who don t consider using it in the future mainly consider one-on-one training (41%), short video clips demonstrating how to use a tool (34%) and online course with facilitation by e-learning support staff (32%). Table 4 Previous or envisaged future use of clickup training and support opportunities I ve used this service in the past I would like to make use of this service in the future Face-to-face workshops 45.6% (197) 25.7% (111) 25.2% (109) One on one training 14.4% (62) 34.7% (150) 40.5% (175) Online tutorials 12.3% (53) 50.9% (220) 25.7% (111) Online course with facilitation by EI 6.3% (27) 44.4% (192) 33.3% (144) ICL 2010 Proceedings Page 1070 I have never used this service before and don t think I would be using it in the future 6

7 Fact sheets/ step-by-step sheets with instructions 16.9% (73) 53.7% (232) 21.1% (91) (in words) on where to click Short video clips demonstrating how to use a 3.2% (14) 49.1% (212) 34.5% (149) tool Concise training sessions similar to video 2.3% (10) 53.2% (230) 32.2% (139) conferencing that I can attend on my own computer in my own office Customised sessions based on the needs of my 8.8% (38) 57.9% (250) 22.2% (96) own academic department The clickup help website % (117) 40.3% (174) 19.9% (86) e-support office 28.9% (125) 35.4% (153) 21.5% (93) Instructional designers for advice on the use of 25.5% (110) 42.1% (182) 21.1% (91) clickup Assistance from another academic staff member in the use of clickup 32.4% (140) 35.4% (153) 22.0% (95) 4 Discussion The preferences to use new technologies as indicated by faculty appeared to correlate with the innovation adoption curve of Rogers [6]. From Figure 1 it is clear that the majority of respondents form part of the late majority group ( I use new technologies when most people I know do ), while only a small number of respondents are part of the laggards group ( I am usually one of the last and I am skeptical ). The right side of the figure indicates that the early adopters ( I love new technology ) and the early majority ( I like new technology. ) also constituted a large number of the respondents. The daily and regular use of the LMS by the majority of the respondents (62%) indicates that the majority of respondents have attained a level of use of the LMS as a routine practice. According to the Level of Use construct of Hall s implementation bridge [2] the next step in the level of use for these users would be to move to the Refinement level, where users modify their use of the LMS to increase or improve the outcomes they wish to attain. The support strategy of EI should thus change course to assist academic staff in this new phase. Although the majority of users are using the system routinely, there are other users on all the different levels of use described by Hall who will also need assistance with their personal progression through the levels of use. Lecturers perceive the educational value added by the LMS mainly with regards to content delivery and administrative functions (Table 1). This gives an indication that the users are at Stage 3 of the Stages of Concern (SoC) construct within Hall s implementation bridge [2]. Stage 3 of the SoC (Management) implies that the user focuses on the best ways to apply information and resources to enhance efficiency, and on problems related to organising, managing, and scheduling and handling time demands. To reach the next stage (Consequence) in the SoC, users attention will have to move toward appreciating the impact their teaching and learning strategies within the LMS have on student learning. The factors that most significantly impact on lecturers not using more functions are a lack of knowledge of more features available in the system, as well as personal time constraints (Table 3). These findings correlate with a number of other studies that identified the lack of time and time for training as one of the major obstacles in the adoption of an LMS [7,8,9]. Both these factors will have to be addressed by EI. It is evident that EI will have to raise awareness of the functions available within the system and the impact it might have on ICL 2010 Proceedings Page

8 teaching and learning. Secondly, training and support strategies need to be adapted to suite the academic staff s limited time. The training workshops presented by EI mainly focus on technical where to click strategies, and only occasionally pedagogical ideas and cases are presented to illustrate the use of the tools. Furthermore, the use of fact sheets (step by step how to documents) also underscored the mechanical use of the LMS. This focus has directed the academic staff s attention to know how to use the tools technically, but may have hindered the sound pedagogical implementation of the system to its fullest potential. Birch and Burnett s framework [4] that indicates that the individual s history will impact on the future use of e- learning, and the study by Samarawickrema and Stacy [5] that emphasizes the motivation of staff and their approaches to change and learning, suggest that distinct and strong guidance will be needed to change the current level of use within the university. The indications from users that they would like training strategies that address their departmental needs, may provide opportunities to develop customised strategies for higher levels use. Opportunities that enable academic staff to undergo just-in-time and just-for-me training need to be enhanced and expanded. The way in which questionnaire items were developed, namely only using issues raised during focus groups, may have limited the extent to which pedagogical issues could have been explored in this study. These could be addressed in future studies. Further investigations of the underlying educational principles, as well as the personal beliefs of academic staff members concerning the application of these principles in the e-learning environment are also required. The fact that significant more non-users than users do not explore and do not make us of more features within the system due to the problems or limitations within the system (Table 3) needs to be explored in greater depth to determine whether these perceptions are based on personal experience or other factors. 5 Conclusions If Hall s [2] questions are used to determine the dissemination of the LMS at the University of Pretoria, it is evident that the system is being used by a substantial numbers of lecturers. The main reasons why lecturers make use of the system are that it adds value to their educational practice and saves them time. The level of use is however not optimal as mainly administrative and content dissemination functionalities are being used, with much lower usage of the communication and assessment functionalities available in the system. It is evident that the Department for Education Innovation will have to adapt their strategies to shift the use of the LMS to a higher level thereby transforming the current teaching and learning model. References: [1] Coates,H.; James, R. and Baldwin, G.: A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, Vol 11, pp 19 36, [2] Hall, G.E.: Technology s Achilles Heel: Achieving High-Quality Implementation. Journal of Research on Technology in Education, Vol. 42, No. 3, pp , [3] Straub, E. T.: Understanding technology adoption: Theory and future directions for informal learning. Review of Educational Research, Vol. 79, No. 2, pp , ICL 2010 Proceedings Page 1072

9 [4] Birch, D and Burnett, B.: Bringing academics on board: Encouraging institution-wide diffusion of e-learning environments, Australasian Journal of Educational Technology, Vol. 25 No.1, pp , [5] Samarawickrema, G. and Stacey, E.: Adopting Web-Based Learning and Teaching: A case study in higher Education. Distance Education, Vol. 28, No. 3, pp , [6] Rogers, E.M.: Diffusion of innovations. New York, N.Y.: Free Press, 1995 [7] Chizmar, J. F. and Williams, D. B.: What do faculty want? Educause Quarterly, Vol. 24, No.1, pp , 2001 [8] Becta (British Educational Communications and Technology Agency) (June, 2004). A review of the research literature on barriers to the uptake of ICT by teachers. < Retrieved June 5, [9] Al-Senaidi, Lin L and Poirot, J.: Barriers to adopting technology for teaching and learning in Oman. Computers & Education, Vol 53, pp , Author(s): Detken Scheepers, Mrs. University of Pretoria, Department for Education Innovation University of Pretoria 0002 detken.scheepers@up.ac.za Rhena Delport, Prof University of Pretoria, Department of Family Medicine University of Pretoria 0002 rhena.delport@up.ac.za 9 ICL 2010 Proceedings Page 1073