Current State and Trends in Learning Analytics

Transcription

1 Current State and Trends in Learning Analytics Dragan January 22, 2016 Open University of Hong Kong Hong Hong

2 Educational Landscape Today Non-traditional students Redefining the role of universities Changing labor market

3 Learning at scale

4 Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences,

5 Feedback loops between students and instructors are missing/weak!

6 LEARNING ANALYTICS

7 Student Information Systems Learners Educators Learning environment

8 Mobile Student Information Systems Networks Learners Educators Search Learning environment Blogs Videos/slides

9 Mobile Student Information Systems Networks Learners Educators Search Learning environment Blogs Videos/slides

10 Learning Analytics What? Measurement, collection, analysis, and reporting of data about learners and their contexts

11 Learning Analytics Why? Understanding and optimising learning and the environments in which learning occurs

12 Questions explored Pass/Fail, Retention Concept understanding Learning motivation/engagement Learning strategy and metacognition Learning dispositions Graduate qualities Learning experience Satisfaction, community

13 Growing Interest

14

15

16

17 CASE STUDIES

18 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Year 1 Year 2 Year 3 Year 4 Course Signals No Course Signals Student retention Arnold, K. E., & Pistilli, M. D. (2012, April). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp ).

19 Can teaching be improved? Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4),

20

21 Patel, M. S., Asch, D. A., & Volpp, K. G. (2015). Wearable devices as facilitators, not drivers, of health behavior change. The Journal of the American Medical Association, 313(5),

22 Wright, M. C., McKay, T., Hershock, C., Miller, K., & Tritz, J. (2014). Better Than Expected: Using Learning Analytics to Promote Student Success in Gateway Science. Change: The Magazine of Higher Learning, 46(1),

23 Wright, M. C., McKay, T., Hershock, C., Miller, K., & Tritz, J. (2014). Better Than Expected: Using Learning Analytics to Promote Student Success in Gateway Science. Change: The Magazine of Higher Learning, 46(1),

24 INSTITUTIONAL ADOPTION: CURRENT STATE

25 Very few institution-wide examples of adoption

26 Analytics Framework Stage 1: Extraction and reporting of transaction-level data Stage 2: Analysis and monitoring of operational performance Stage 3: What-if decision support (such as scenario building) Stage 4: Predictive modeling & simulation Stage 5: Automatic triggers and alerts (interventions) Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management information and technology in higher education (Vol. 8). Educause.

27 ~70% institutions in phase 1 Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management information and technology in higher education (Vol. 8). Educause. 305 institutions, 58% at stage 1, 20% at stage 2 Yanosky, R. (2009). Institutional data management in higher education. ECAR, EDUCAUSE Center for Applied Research.

28 Interest in analytics is high, but many institutions had yet to make progress beyond basic reporting Bichsel, J. (2012). Analytics in higher education: Benefits, barriers, progress, and recommendations. EDUCAUSE Center for Applied Research.

29 Sophistication model Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Productivity of the Higher Education Sector - Policy and Strategy for Systems-Level Deployment of Learning Analytics. Canberra, Australia: Office of Learning and Teaching, Australian Government. Retrieved from

30 Sophistication model Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Productivity of the Higher Education Sector - Policy and Strategy for Systems-Level Deployment of Learning Analytics. Canberra, Australia: Office of Learning and Teaching, Australian Government. Retrieved from

31 Current state Benchmarking learning analytics status, policy and practices for Australian universities

32 Senior management perspective

33 Senior management perspective

34 Solution-driven approach Bought an analytics product. Analytics box ticked!

35 Lack of data-informed decision making culture Macfadyen, L., & Dawson, S. (2012). Numbers Are Not Enough. Why e-learning Analytics Failed to Inform an Institutional Strategic Plan. Educational Technology & Society, 15(3),

36 Researchers not focused on scalability

37 Learning from the successful examples

38 An institutional learning analytics vision Tynan, B. & Buckingham Shum, S. (2013). Designing Systemic Learning Analytics at the Open University. SoLAR Open Symposium Strategy & Policy for Systemic Learning Analytics.

39 What s necessary to move forward?

40 DIRECTIONS

41 Data Model Transform Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

42 Data Model Transform Creative data sourcing Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

43 Social networks are everywhere Gašević, D., Zouaq, A., Jenzen, R. (2013). Choose your Classmates, your GPA is at Stake! The Association of Cross- Class Social Ties and Academic Performance. American Behavioral Scientist, 57(10),

44 Data Model Transform Creative data sourcing Necessary IT support Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

45 Awareness of limitations and challenging assumptions Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. (in press). Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics, 2(3).

46 Data Model Transform Question-driven, not data-driven Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

47

48 Learning analytics is about learning Gašević, D., Dawson, S., Siemens, G. (2015). Let s not forget: Learning analytics are about learning. TechTrends, 59(1),

49 Once size fits all does not work in learning analytics

50 Learning context Instructional conditions shape learning analytics results Gašević, D., Dawson, S., Rogers, T., Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of course-specific technology use in predicting academic success. The Internet and Higher Education, 28,

51 Learner agency More time online does not always mean better learning Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., Adesope, S. (2015). Analytics of Communities of Inquiry: Effects of Learning Technology Use on Cognitive Presence in Asynchronous Online Discussions. The Internet and Higher Education, 27,

52 Data Model Transform Participatory design of analytics tools Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

53 Data Model Transform Participatory design of analytics tools Analytics tools for non-statistics experts Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

54 Visualizations can be harmful Corrin, L., & de Barba, P. (2014). Exploring students interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the ascilite 2014 conference (pp ). ascilite.

55 Data Model Transform Participatory design of analytics tools Analytics tools for non-statistics experts Develop capabilities to exploit (big) data Barton, D., & Court, D. (Oct 2012). Making Advanced Analytics Work for You. Harvard Business Review, 79-83,

56 Marr, B. (Oct 2015). Forget Data Scientists - Make Everyone Data Savvy,

57 Are we ready to act on analytics?

58 Are we ready to act on analytics? What to do if we detect deficit models in our practice? Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., Hatala, M. (2015). Learning at distance: Effects of interaction traces on academic achievement. Computers & Education, 87,

59 Are we ready to act on analytics? How do we deal with performance-oriented culture? Jovanović, J., Pardo, A., Gašević, D., Dawson, S., Mirriahi, N. (2015). Dynamic analytics of learning in flipped classrooms. Manuscript in preparation.

60 LA idealized systems model Colvin, C., Rogers, T., Wade, A., Dawson, S., Gasevic, D., Buckingham Shum, S., Nelson, K., Alexander, S., Lockyer, L., Kennedy, G., Corrin, L., Fisher, J. (2015). Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement. Australian Goverement s Office for Learning and Teaching.

61 Rapid Outcome Mapping Approach (ROMA) Macfadyen, L. P., Dawson, S., Pardo, A., & Gasevic, D. (2014). Embracing big data in complex educational systems: The learning analytics imperative and the policy challenge. Research & Practice in Assessment, 9(2),

62 FINAL REMARKS

63 Embracing complexity of educational systems

64 Capacity development Multidisciplinary teams in institutions critical

65 Ethical and privacy consideration Development of data privacy agency Prinsloo, P., & Slade, S. (2015). Student privacy self-management: implications for learning analytics. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp ). ACM.

66 Sclater, N. (2014). Code of practice for learning analytics: A literature review of the ethical and legal issues.

67 Development of analytics culture Manyika, J. et al. (2011). Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute,

68

69 Many thanks!