What Makes a Learning Analytics Dashboard Successful?

Transcription

1 What Makes a Learning Analytics Dashboard Successful? Short Description Despite growing interests in Learning Analytics Dashboard (LAD), few studies have investigated factors that determine successful LAD.This study investigated structural relationships among design and learner factors for a successful LAD. The data collected from 273 university students enrolled in one online course over two consecutive semesters was analyzed using structural equation modeling. The result indicates that boosting learners understanding and reflective use is needed to promote their perceived effectiveness and behavioral changes. Abstract Introduction Learning analytics dashboard (LAD) is an emerging tool as it supports learners self-regulated learning, engagement, and academic achievement in online learning contexts (Corrin & Barba, 2014). Although there is no general consensus yet on the definition of LAD, recent research recognizes it as a tool to promote learners awareness of, and reflection on their learning process by displaying student learning information (Santos, Govaets, Verbert, & Duval, 2012). LAD provides various modules for learners to monitor their learning progress mostly via visualization of real-time data (Verbert et al., 2014). Researchers have focused on effective visualization of data and suggested a way to provide interventions based on tracked data (e.g., Corrin & Barba, 2014; Duval, 2011). Despite the exponential growth being made in this field, few studies have investigated factors that determine successful LAD. Prior research is mostly limited to testing usability and usefulness, not addressing learners learning process with LAD (Verbert et al., 2014). Considering that the ultimate goal of use of LAD is to stimulate behavioral changes based on self-monitoring, it is important to examine what factors lead to the positive outcomes. Displaying information alone does not necessarily guarantee successful use of LAD. Recent research reports challenges that learners encounter with regard to low usefulness and interpretability of LAD (Corrin, Kennedy, & Mulder, 2013). The ultimate goal of this study is to examine what factors engage learners with LAD. We investigated structural relationships among design and learner factors that are related to learners' perceived effectiveness and behavioral changes. The findings provide suggestions for effective LAD. Design and learner factors This study included the following variables to investigate structural relationships among design and learner factors. Information usefulness Given LAD is to provide learners with useful information, it is important to minimize a gap between perceived information usefulness and given information (Corrin & Barba, 2014). We cannot expect learners to be motivated to use LAD without meaningful information to them. It is known that perceived usefulness predicts learners genuine interest and seek for achievement (Liem, Lau, & Nie, 2008).

2 Effective visualization As mentioned previously, visualization is key a feature of LAD. There has been an effort to find an effective way to visualize learners online behaviors relevant with learning activities (e.g., Bakharia & Dawson, 2011; Coffrin, Barba, Corrin, & Kennedy, 2014; Duval, 2011; Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). Ease of use How easily learners can use LAD affects learners' inclination to be attached to LAD. Previous studies emphasize the importance of usability as reflecting the affordance that leads to behavioral engagement and perceived effectiveness (Govaerts, Ververt, Duval, & Pardo, 2012). Understanding level Even if LAD provides a large amount of information to learners, successful use depends on the degree of which learners understand the information to improve their learning. In this study, researchers considered the understanding of displayed information as mediating the suggested factors. Reflective use Learning analytics approach has been employed to utilize learners data for encouraging their reflection process, which ultimately leads to persistence in learning (Li, Dey, Forlizzi, Höök, & Medynskiy, 2011). In many cases, LAD is intended to promote learners awareness and reflection by providing an overview of their individual progress as well as performance compared to peers. This variable was to see if learners actually used LAD for reflective purpose (e.g., self-monitoring and reflection on progress). Perceived effectiveness Perceived effectiveness is a strong indicator of successful LAD, based on learners perception. This factor has been widely used in prior literature (Ali, Hatala, Gašević, & Jovanović, 2012) as it reflects learners positive attitudes and engagement. Behavioral engagement We included this variable to see the factors suggested above lead to actual behavioral changes in learning. Research hypotheses Figure 1 shows the hypothesized model for this research.

3 The following hypotheses were tested (See Figure 2).

4 Method In the current study, we identified structural relationships among the factors suggested above. We collected data from 273 university students enrolled in one course taught in two consecutive semesters. The course was 100% online and all the participants were required to use a LAD. The LAD developed for this research provided a self-monitoring widget where participants could check their log information such as total login time, frequency, and login interval regularity. The information was displayed either in graphs or charts (See figure 3). The participants were able to see where they fall in the chart, compared to their peers. Structural equation modeling was performed to test the research hypotheses. Results Table 1 shows statistics for the structural model. The result indicates the structural model has a good fit to the collected data.

5 Table 2 shows the result of hypotheses test. All hypotheses were all accepted. Discussion & Closing The result indicates that useful information leads to the reflective use of LAD. To facilitate learners reflective action such as self-monitoring, we need to take into account what information learners recognize as useful. This result supports the expectancy-value theory that recognizes attainment value as a determinant of intrinsic motivation (Eccles, 1983). Visualizing information in an organized and clear way is also important factor as learners understanding level is affected by correct interpretation of information, which in turn leads to reflective use of LAD. It was found that both understanding level and reflective use mediate between design factors and effectiveness. That is, we should not ignore the importance of reflective use and understanding. For example, instructors need to be aware of whether learners clearly understand what given information implies, and how they use it for their own improvement. In summary, we need to pay attention to learning process rather than just letting them wandering around LAD. Appropriate instructional strategies are also needed since learners perceptions and behavioral changes are determined by how well they understand and use given information. We hope this study contributes to the field of learning analytics by providing directions for designing and managing LAD.

6 <Reference> Ali, L., Hatala, M., Gašević, D., & Jovanović, J. (2012). A qualitative evaluation of evolution of a learning analytics tool. Computers & Education, 58(1), Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp ). ACM. Bakharia, A., & Dawson, S. (2011). SNAPP: a bird's-eye view of temporal participant interaction. In Proceedings of the 1st international conference on learning analytics and knowledge (LAK) (pp ). ACM.. Coffrin, C., Corrin, L., de Barba, P., & Kennedy, G. (2014). Visualizing patterns of student engagement and performance in MOOCs. In Proceedins of the Fourth International Conference on Learning Analytics and Knowledge (LAK) (pp ). ACM. Corrin, L., & de Barba, P. (2014). Exploring students interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the ascilite 2014 conference. Dunedin, NZ. Corrin, L., Kennedy, G., & Mulder, R. (2013). Enhancing learning analytics by understanding the needs of teachers. Electric Dreams. Proceedings ascilite, Duval, E. (2011). Attention please!: learning analytics for visualization and recommendation. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK) (pp. 9-17). ACM. Eccles, J. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives: Psychological and sociological approaches (pp ). San Francisco, CA: W. H. Freeman. Gašević, D., Dawson, S., & Siemens, G. (2015). Let s not forget: Learning analytics are about learning. TechTrends, 59(1), Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012). The student activity meter for awareness and self-reflection. Paper presented at the CHI'12 Extended Abstracts on Human Factors in Computing Systems (pp ). ACM. Li, I., Dey, A., Forlizzi, J., Höök, K., & Medynskiy, Y. (2011). Personal informatics and HCI: Design, theory, and social implications. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, CHI EA 11 (pp ). New York, NY: ACM. doi: / Liem, A. D., Lau, S., & Nie, Y. (2008). The role of self-efficacy, task value, and achievement goals in predicting learning strategies, task disengagement, peer relationship, and achievement outcome. Contemporary Educational Psychology, 33(4), Santos, J. L., Govaerts, S., Verbert, K., & Duval, E. (2012). Goal-oriented visualizations of activity tracking: a case study with engineering students. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp ). ACM. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Van Assche, F., Parra, G., & Klerkx, J. (2014). Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing, 18(6),