Business Intelligence in e-learning

Transcription

1 Business Intelligence in e-learning Julija Lapuh Bele, Darko Bele, Rok Pirnat, Vedran Anžin Lončarić B2 d.o.o. Ljubljana, Slovenia {julija.bele, darko.bele, Julija Lapuh Bele, Darko Bele Ljubljana School of Business Ljubljana, Slovenia {julija.bele, Abstract The goal of the study was to discover which built-in tools in business analytics were suitable for implementation in the ecampus LMS from the perspective of various stakeholders. A second objective was to prepare data and built-in analytics, and to carry out and evaluate a pilot implementation of LMS with data analysis tools. When e-learning courses are conducted in the context of companies, managers are primarily interested in the effects of the e-course on the skill sets of their employees. They need to measure the return on investment in employee training courses. However, data collected before, during, and after e-learning courses may also be useful for other stakeholders: learners, instructors, managers of human resources departments, and professionals in the companies that provide e-learning. Each of the various stakeholders, depending on their specific role, require different kinds of analyses and reports in order to improve processes and quality. Keywords e-learning; business intelligence; BI in e-learning; learning analytics; data visualisation I. INTRODUCTION Since the term business intelligence (BI) was first introduced, the set of techniques and tools for the acquisition of meaningful and useful information from a large amount of data has evolved rapidly. Information-based decision-making is used in a number of business areas, including human resource management (HRM). According to Gartner [3], advanced, pervasive, and invisible analytics are one of the top technology trends in 2015 and will be for years to come. Every modern application needs to be an analytic application. Therefore LMS will also evolve into an analytic system of applications. The LMS ecampus is used for conducting e-learning courses in private and public workplaces. Managers are interested in understanding the effects of their employees learning and measuring the return on investment in employee training courses. In addition, data collected before, during, and after e- learning courses might also be useful for instructors or mentors, for the professionals in the human resources department, and for department managers. Therefore, the goal of our study was to find out which builtin tools of business analytics are most suitable for various stakeholders in the implementation of the LMS ecampus. A secondary objective was to prepare data and built-in analysis, and to carry out a pilot implementation of LMS with the tools for data analysis. First, instructors and various types of managers (including HRM professionals, heads of divisions, and CEOs) were asked what data and built-in analytics would help them in their decision-making processes. Second, a pilot version of the LMS with BI tools was developed in order to make an evaluation. All stakeholders have been asked to express their opinion on the usefulness and functionality of the solutions. According to users opinions and suggestions, the LMS ecampus will be upgraded to the level of an analytic system. II. BUSINESS INTELLIGENCE IN E-LEARNING A. Business Intelligence in LMS The main goals of LMS are to manage e-learning processes and to support communication among learners. An effective and efficient LMS helps stakeholders in the learning process to set goals, distribute e-learning content, track learning, and analyze e-learning processes and reporting. Researchers in the field of e-learning commonly use LMS to focus on learner-instructor and learner-content interactions. However, there are other stakeholders in the process, such as the management of the e-learner provider, the management of the client company, and the HRM department also require data and data analytics to improve performance. BI is closely linked to knowledge management, which integrates the sub-processes of knowledge creation, enabling access to knowledge, and use of knowledge [8]. Access to knowledge is made possible by the appropriate technology, which also contributes substantially to knowledge management in a company. The definition of BI has changed over time. Today, we usually understand BI as a tool for supporting the selection, presentation, and analysis of data. BI is a set of processes, technologies, and tools that are necessary to transform data into information, information into knowledge, and knowledge into plans for implementing business measures that increase efficiency and profitability. BI includes the concepts of the data warehouse, business-analytical tools and content, and knowledge management [9]. The purpose of BI in e-learning is to provide users with information that will help them achieve greater success and efficiency. In order to successfully organize and implement e-learning, we need a range of stakeholders (learners, instructors, e-learning content authors, e-learning providers, clients). All activities are

2 intended for the individual learners, to create more efficient learning processes, and to improve their skills. In order to achieve these goals, it is necessary for the learning environment to guarantee the following functionalities: usable, user-friendly platform for e-learning, high-quality e-content and e-courses, support services for e-learning. All the users of e-learning platforms will pursue strategies for achieving the specific goals associated with their particular role or mission described in Table 1. Learner Instructor Author User E-learning provider Client TABLE I. USER S ROLE /MISSION Role / mission efficient learning, improvement of skills, acquisition of knowledge, certification provide learners with an excellent learning experience, promote and foster e-learning, supervise the flow of learning, understand the learners, create quality e-content measure the success of e-learning courses and the level of learner satisfaction, analyze activities and gained data, improve processes evaluate the efficiency of the learning process in the framework of defined goals and resources used B. Data Mining in e-learning Data mining is the process of analyzing data from different perspectives and summarizing it into useful information that can be used to increase revenue, cuts costs, or both. Data mining is used to extract information - the "gold" - from the larger data "ore" [1]. Data mining is the process of finding correlations or patterns among dozens of fields in large databases and warehouses. Data mining in an educational setting uses methods such as forecasting, combining into groups, mining contacts, distilling data for human judgment, and discovering patterns and models [2]. C. Learning Analytics Learning analytics researches possibilities for using intelligent methods and procedures with the purpose of improving the learning process from the perspective of learners, instructors, and institutions that provide educational services, and also from the perspective of clients when the purpose is training or providing educational services for employees of a company. The research generally relies on data mining in education, and includes concepts and techniques from information science and sociology, including computer studies, statistics, psychology, and pedagogy. The goal is to understand the components of the learning process and thus help to define the goals of all users [6]. The expression learning analytics first appeared in 2009 in Horizon Reports [5], a publication that presents emerging technologies and fields of research. It is expected that in the coming years, learning analytics and other developments and achievements in this field, will have a great influence on learning processes at all levels of education and will equip instructors with computer support systems that can be adapted to the needs and potentials of each learner. The fundamental goal of learning analytics is not computer generated or automated responses to the activities of learners, but rather the communication of interesting findings to those who create learning situations (mentors, authors of e-content, and educational institutions). These findings will deal with subjects such as the easier adaptation of learning content, mediation with high-risk learners, and the creation of information feedback systems about learners [14]. The methods using in learning analytics are varied. In addition to those that use data mining in education (forecasting, forming groups, mining contacts, distilling data for human judgment, and discovering patterns and models), the following two methods should also be included: Social network analysis: deals with the analysis of interaction between learners-learners or learnersinstructors on the basis of which it is possible to conclude which learners do not participate in discussions (for example, on social networks or forums posted by instructors), and which learners tend to stick to themselves and perhaps do not show enough interest in the subject. It would also be possible to search out the learners who participate in many different discussions and make connections with other learners and instructors. Social or attention metadata: provides insight into what learners are doing, at what activities individual learners are most efficient, which activities they find most interesting, etc. Similar to data mining in education, it is important that we provide results in a clear manner, using, for example, methods for distilling data for human judgment. Clear and comprehensible presentation is extremely important because all measures for helping learners will be derived from it. Only a clear presentation of status, pitfalls, and progress that is understandable to both learner and instructor can facilitate decision-making about the continued learning process and minimize the potential for error. The main uses of learning analytics are monitoring progress, forecasting the success of the learner, and early detection of potential problems. These in turn allow for timely intervention with learners who have a greater likelihood of not completing a subject or a year of study [6], [11]. In sum, learning analytics performs the following functions: analyzes various data produced by learners, or are somehow connected to learners, scans for ways to monitor the activities and progress of learners, and to forecast their success, presents data in a way that enables both instructor and learners to intervene and change the course of continued studies.

3 D. Visual Analysis of Data The visual analysis of data combines demanding computing methods with advanced graphics, and therefore takes advantage of the capability of the human brain to quickly perceive patterns and trends in complex graphic presentations [6]. The visual analysis of data is comprised of the following two phases: presentation of data, discovery of patterns and models in the data. The first phase is computer-supported. It is important that data be visualized in a way that allows the detection of patterns. In the second phase, the user interactively studies the presented data, changing the perspective, zooming in and zooming out, marking selected groups of data, monitoring how data characteristics change over time. Numerous factors have an impact on data visualization, from the choice of text to what type of graphic presentations are used. Despite many complex methods for analyzing and presenting analyses, the best-designed graphics are simple, direct, and efficient means of presenting information [12]. E. Tracking the Learning Process The easiest way to achieve basic information about a learner's current status is a through a presentation of statistical data about the learning process. It is essential that the learner can also track the flow of his or her own learning process, which means that data must be presented in a simple and understandable manner. Care must be taken that the data have a clear connection to the learning process and that they are not too detailed, as the amount of data the system can select is enormous. For most stakeholders information comes in the form of descriptive statistics: average grades of the learner or group of learners; e-course participation; grades of learners for individual courses, content, or assignments; performance on knowledge evaluations; most frequent errors in problem solving on tests; studied material; most popular websites visited; statistical reports about the use of certain forums and tools for social networking, etc. The reports should be informative, clear, and easy to understand. An overview of the activities of the use of LMS that informs the instructor about who logs into the system, for how long, how much time each user spends on certain pages, and what pages they visit is provided by Pahl and Donnellan [10]. From this data, an instructor can quickly conclude which learners are the least active so that the instructor can encourage them and perhaps find more interesting activities. It also gives insight into which pages or activities attract the most users. Instructors can then examine the reasons for the high versus low level of visits and take appropriate measures in shaping the learning program. III. BI IMPLEMENTATION IN THE ECAMPUS SYSTEM In order to achieve educational goals, it is necessary to ensure that the learning environment provides a functionally appropriate and user-friendly platform for high-quality e- learning as well as support services for e-learning. From the standpoint of BI, we wanted above all to support pedagogical services in e-learning, although the data we gathered about the functioning of the system are interesting for other types of analyses as well. A. Identifying the Needs of System Users We identified the BI-related needs of the various stakeholders on the basis of interviews that we conducted with 105 learners, instructors, authors of e-content, and managers of both educational service providers and companies that use ecampus LMS. Users need a range of information. While they may need different information, most nevertheless share the desire for information that is simple and easy to access and act upon. 1) Learner Based on the interviews, we discovered that learners need the following information: progress report on independent learning using e-content (for example, the share of completed material), overview of success on graded learning activities (for example, written assignments, tests to evaluate knowledge, etc.), overview of status (completed learning activities, learning activities in the process of completion, etc.), overview on the flow of access to e-content (in the case that content is distributed at specific times). 2) Instructor/Mentor In order to conduct a high-quality and efficient e-course, the instructor/mentor needs the following: analytics with visualization of data, tools that allow action based on analytic reporting (for example, communicating with selected users). The instructor needs information about how learners are performing during various phases of the learning process in order to formulate appropriate motivational strategies for individual users and groups of users. Key information includes, for example, the share of completed material, grades on tests and written assignments, the number of postings on shared learning platforms (forums, social networks), and the degree of completion of the e-learning course. 3) Manager at Client Company When e-learning takes place within a company or for a company, it is often the case that not only the instructor, but also a professional manager or supervisor, carries out and has an interest in the process. An instructor may run the e-course and lead employees through the learning process, but participants have their own superiors in the work environment and sometimes even work mentors. If participants are receiving training in the work place, the employer or supervisor is usually interested in knowing whether the time spent on the course was a good investment and what were the results of the course (acquired skills, knowledge, and participant satisfaction). In companies that train employees, the department of human resources (HRM) usually manages development services. Therefore there are two types of managers involved: direct managers of the employees who are participating in the educational process and are interested in the additional skills and competences acquired by employees, and general managers or

4 HRM managers who do not directly participate in the process and are interested not only in acquired skills but also on the return on investment in education. Managers require graphic visual reporting that presents all necessary information, since they sometimes do not know how to access the system themselves and do not have the time to sort through large amounts of data. Informing managers often has a decisive significance on the success of the educational process, as the employer generally has clear goals in terms of educational results, time during which the course runs, and how this impacts work performance. Managers need a simple overview of the achievements of the learning program according to subject and groups of participants that report to the manager s department. The overview should also allow for simple communication with participants and groups of participants on the basis of the conclusions. 4) Manager in Educational Institution In terms of the institution that runs the e-learning program or the company education center, those who lead the processes are generally interested in development and ongoing quality improvement. With data mining and BI analysis, we can provide these professionals with ongoing information throughout the entire educational process. For this purpose, it is necessary to establish a data warehouse, as processing and analysis on a transactional base slows the system and provides the user with less data than the system recorded during the time of operation. Using the data warehouse we created for the ecampus, we can provide either "ad-hoc" reports with Excel (PowerPivot) or other focused tools that allows "ad-hoc data mining" analysis. B. Implementation of Learning Analytics in the ecampus Information is provided to the user in the form of tables and charts. Presentations are generated using various systems of visualization and interactive graphs and charts, mostly donut, bar and bubble charts. Clicking on a specific part of the charts reveals more detailed information about that section. Clicking on the white section of the donut chart switches off the filters. Two-dimensional data are represented with bar charts. Threedimensional data are represented using bubble charts. E-course is represented by e-classroom in ecampus LMS. 1) Learner The learner is the most important stakeholder in the e- learning process, as the entire process takes place because of the learner, and is thus learner-centered. Because of this, most LMS e-learning formats provide individual learners with an overview of learning achievements, activities, and results. Figure 1 provides an example of a visual report that shows the learner s data on learning from e-contents. In addition to providing a visually pleasing overview of learning activities and success, by clicking on various icons in the report, learners can post information on social networks such as LinkedIn, Facebook, and Twitter. If, for example, a learner participates in a course and acquires a certificate, this information can be posted on the learner s LinkedIn profile. 2) Professional Manager of Client Company When considering the perspective of the professional manager or supervisor, we starting with the following identified requirements: Figure 1: Visual report on use of e-learning materials

5 Information for managers should be easily accessible, understandable, and comprehensive. The information that the manager is most likely to need should be presented first. Managers are usually most interested in whether employees completed the assigned tasks, how much time was needed to complete the course, and whether specific skill or certifications were acquired. An analysis of the learning in e-classrooms (by activity) should be available to managers, and especially an analysis of learning from e-content. The analysis allows for a simple filtering of users regarding the success of certain activities (for example, learning from specific e-content). For simplicity sake, we created three groups that are shown in the donut chart (Figure 1). The standard (for example, 70%) may be set by users i.e. managers or mentors/instructors. Inactive users users who did not complete the standard level of activities. Active users users who completed the standard level of activities, but have not yet completed all activities. Successful completion users who have completed all activities. The analysis is available for individual e-content, individual activities from e-course (e-classroom in ecampus LMS), and for the entire e-course. Managers are often interested in the over-all success of e-courses. Figure 2 shows an example of a report designed for managers. The system includes formulas according to which success in e-learning is calculated. Individual activities in the course have a value from 0 to 5. Given the sum of values of all activities, a weighted average of individual activities is calculated. The sum of the averages is 1 or 100%. Activities can be required, recommended (less, medium, or more recommended), or not required. The instructor defines the values of activities. In cases where the instructor does not define the relative importance of activities, they have equal value. Although certain activities may begin in a later phase of the course, our goal is to monitor the current activities of users. Therefore our reports and calculations reflect the impact of current activities. The impact of current activities is defined as the relationships between current activities and the sum impact of current activities. If the course is designed as self-paced learning, there is no need for the e-classroom in ecampus LMS. In such cases, the manager can choose to view e-content and related analyses. With the help of a chart, a manager can filter employees by their status in a given learning phase. In practice, this means that the manager can choose the red, yellow, or green part of the interactive chart. By choosing a segment on the chart, additional information and people appear along with certain data. The manager can also make use of tools of direct communication, for example, by sending individual learner motivational messages. Figure 2: Report for the manager

6 3) HRM The HRM department needs an entire overview of education in the company. The perspective of HRM is similar to that of the manager with one important difference. For the purpose of HRM, not just the individual is important, but the success of the whole department. Communication takes place with the head of the department, not with the individual. Therefore, more detailed statistics are made available to HRM. This function allows the export of data in csv form. Csv file can be imported in Excel where tools for BI analysis are available. At the conclusion of the course, HRM services and company management require different reports indicating the success of the educational program or providing other information required for evidence and on the basis of which improvements can be made. The manager can also be sent a more comprehensive report about the completed course formatted as a PDF file. 4) Instructor/Mentor The instructor/mentor needs analytics with visualization of data and tools that allow action based on analytic reporting. The reports represent how a group or individual learners are performing during various phases of the learning process. The information is available on particular e-content, e-course (named e-classrom in ecampus LMS). or particular activity in the e- course. Key information includes, for example, the share of completed material, grades on tests and written assignments, the number of postings on shared learning platforms (forums, social networks), and the degree of completion of the e-learning course. Figure 3 shows an example of a report designed for instructors. The instructor can click on the chart and find out about the learner represented by a specific bubble. IV. CONCLUSION On the basis of research into the needs of e-learning users, we developed analytic tools in ecampus LMS and created graphic presentations of information that allow various stakeholders to evaluate the success of the learning process on the basis of received information. With state-of-the-art visualization techniques, we succeeded in introducing graphic elements to the functionality that will help users to take desired actions. The BI mission is to direct users toward the right decisions and, consequently, toward appropriate actions that increase efficiency and improve success. Our future research will analyze the usability of the system and measure user satisfaction with offered solutions. ACKNOWLEDGMENT This research was co-financed by the Ministry of Economic Development and Technology of the Republic of Slovenia and the EU European Social Fund. Figure 3: Report for the instructor

7 REFERENCES [1] M. Bienkowski, M. Feng and B. Means. (2012, dec ). Enhancing Teaching and Learning through Educational Data, Mining and Learning Analytics: An Issue Brief. US Department of Education, Washington, D.C, Available: [2] E. Cipolla. (1995, Dec 1). Data mining: techniques to gain insight into your data. Enterprise System Journal. Available: [3] Gartner. (2014, Oct 8). Gartner Identifies the Top 10 Strategic Technology Trends for 2015 (Gartner Symposium ITxpo 2014). Available: [4] M. Jeusfeld, H. Shu, M. Staudt and G. Vossen. Design and Management of Data Warehouses, Workshop DMDW'00, CEUR Workshop Proceedings, [5] L. Johnson, A. Levine, R. Smith and S. Stone. The 2010 Horizon Report, The New Media Consortium, [6] L. Johnson, R. Smith, H. Willis, A. Levine, and K. Haywood The 2011 Horizon Report, The New Media Consortium, [7] B. Knight, D. Knight, A. Jorgensen, P. LeBlanc, M. Davis, Microsoft Business Intellignece 24- Hour Trainer. Wiley Publishing, Inc [8] A. Kovačič, B. Vukšić. (2005). Managament poslovnih procesov prenova in informatizacija poslovanja. GV založba, Ljubljana, [9] D. Loshin. Business Intelligence: The Savvy Manager's Guide, Getting Onboard with Emerging IT. Morgan Kaufmann, [10] C. Pahl, D. Donnellan. Data mining technology for the evaluation of web-based teaching and learning systems, in the Conference on E- Learning in Business, Government and Higher Education, 2000, pp [11] L. Taylor and V. McAleese. (2012, July 18). Beyond Retention: Using Targeted Analytics to Improve Student Success. Available: [12] E. R. Tufte. The Visual Display of Quantitative Information. Graphics Press, [13] C. Zinn, O. Scheuer. Getting to know your student in distance learning contexts, in Innovative Approaches for Learning and Knowledge Sharing, Springer, 2006, pp [14] J. Zupančič, E. Dovgan, B. Filipič and R. Pirnat. Mere uspešnosti v sistemih za e-izobraževanje in Proceedings of the 16th International Multiconference Information Society - IS 2013, Ljubljana, 2013, pp