Introduction to Learning Analytics

Transcription

1 Educa&onal Data Mining workshop, Advanced Analy&cs Ins&tute University of Technology Sydney 22 August 2014 Introduction to Learning Analytics Simon Buckingham Shum Professor of Learning Informatics Director, Connected Intelligence Centre University of Technology Sydney simon.buckinghamshum.net #LearningAnalytics 1

2 learning objective: leave with an expanded vision of analytics better questions to ask in your next analytics conversation 2

3 When the Chancellor announces the adoption of a new economic modelling technique we query the limitations of the model 3

4 we query the limitations of the model 4

5 Siri is smart I was speaking at this event: 5

6 Siri is smart Find code acts in education 6

7 Siri is smart Find code acts in education 7

8 Similarly, when we are confronted with new learning analytics John Behrens (Pearson) LAK13 Panel: Educational Data Scientists: A Scarce Breed 8

9 we should query the limitations of the model John Behrens (Pearson) LAK13 Panel: Educational Data Scientists: A Scarce Breed 9

10 hdps://twider.com/wiswijzer2/status/ Note: check the huge difference between knowing and measuring 10

11 a few quick examples of learning analy5cs 11

12 It s out of the labs and into products: every learning tool now has an analytics dashboard (a Google image search) 12

13 Intelligent tutoring for skills mastery (CMU) In this study, results showed that OLI-Statistics students [blended learning] learned a full semester s worth of material in half as much time and performed as well or better than students learning from traditional instruction over a full semester. Lovett M, Meyer O and Thille C. (2008) The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 14.

14 Purdue University Signals: real time traffic-lights for students based on predictive model Validate a statistical model from: ACT or SAT score Overall grade-point average CMS usage composite CMS assessment composite CMS assignment composite CMS calendar composite Predicted 66%-80% of struggling students who needed help 14 Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007):

15 Purdue University Signals: real time traffic-lights for students based on predictive model Results thus far show that students who have engaged with Course Signals have higher average grades and seek out help resources at a higher rate than other students. Pistilli, M. D., Arnold, K. and Bethune, M., Signals: Using Academic Analytics to Promote Student Success. EDUCAUSE Review Online, July/Aug., (2012). 15

16 Spatial clustering algorithm to provoke reflection 16

17 Posture analysis of fieldwork students 17

18 and many more examples including discourse analytics language technologies to assess the quality of online postings and debate social network analytics graph analytics to assess strength and topics of interpersonal ties epistemic game analytics assessing the degree of professional engagement in authentic project scenarios visualizations to reveal important patterns of tool use over time (see other presentations and tutorials) 18

19 but before we get carried away, let s just pause 19

20 observing, measuring, describing, categorising, classifying, sorting, ordering and ranking). [ ] these processes of meaning-making are never wholly neutral, objective and automated but are fraught with problems and compromises, biases and omissions. Selwyn, N. (2014). Data entry: towards the critical study of digital data and education. Learning, Media and Technology /

21 unpacking the core ques5on for learning analy5cs 21

22 can we tell from your digital profile if you re learning? 22

23 Who? can we tell from your digital profile if you re learning? 23

24 Who? How? With what confidence? After what kinds of training? can we tell from your digital profile if you re learning? 24

25 Who? How? With what confidence? After what kinds of training? can we tell from your digital profile if you re learning? Sourcing which data, with what integrity? 25

26 Who? How? With what confidence? After what kinds of training? can we tell from your digital profile if you re learning? Sourcing which data, with what integrity? What kind of learning? What kind of learner? 26

27 Accounting tools are not neutral accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life. Sage, London. pp

28 In what senses do analy5cs shape the reality they measure? 28

29 How do analy&cs shape educa&on? Poli&cally Analytics reports at the organisational and national levels come with consequences at different scales sometimes punitive, often impacting millions of people. 29

30 How do analy&cs shape educa&on? Ontologically What data, concepts and relationships do the analytics designers seek to model? 30

31 Classification systems provide both a warrant and a tool for forgetting [...] what to forget and how to forget it [...] The argument comes down to asking not only what gets coded in but what gets coded out of a given scheme. Bowker, G. C. and Star, L. S. (1999). Sorting Things Out: Classification and Its Consequences. MIT Press, Cambridge, MA, pp. 277, 278,

32 Visualising attainment and progress 32

33 Which analytics could reflect the progress that Joe has made on so many other fronts other than his SATS? 33

34 Key modelling issue: unit of analysis Discourse analysis: how do machines and humans differ in the way they segment a transcript to make sense of it? Rosé, C. P., & Tovares A. (in press). What Sociolinguistics and Machine Learning Have to Say to One Another about Interaction Analysis. In L. Resnick, Asterhan C., & Clarke S. (Eds.), Socializing Intelligence Through Academic Talk and Dialogue. Washington, D.C.: American Educational Research Association Collective intelligence: If we are shifting from a sole focus on individual accomplishment, to that of group knowledge construction and performance, how do analytics assess changes in a group s knowledge and processes? Chen, B., & Resendes, M. (2014). Uncovering what matters: Analyzing transitional relations among contribution types in knowledge-building discourse. In Proceedins of the Fourth International Conference on Learning Analytics And Knowledge - LAK 14 (pp ). New York, New York, USA: ACM Press. doi: /

35 How do analy&cs shape educa&on? Algorithmically What thresholds, samples, relationships, patterns, etc. do the algorithms encode and seek? On what basis is a recommendation engine proposing interventions? 35

36 governingalgorithms.org Learning Analy&cs In an increasingly algorithmic world [ ] What, then, do we talk about when we talk about governing algorithms? 36

37 governingalgorithms.org A technology or an epistemology? Agency, automa&on, accountabili&es Barocas, S., Hood, S. and Ziewitz, M. (2013). Governing Algorithms: A Provoca5on Piece. Social Science Research Network Paper DOI: hdp://dx.doi.org/ /ssrn A typology of algorithms by genre? The inscrutability of algorithms Norma&vity, bias, values 37

38 Open Learning Analytics: open source algorithmic transparency (at least for those who are literate) no analytics lock-in for educators

39 How do analy&cs shape educa&on? Semio&cally What meaning-making does the representation and interaction design encourage? 39

40 How do analy&cs shape educa&on? By changing the system dynamics outcome intent researchers / educators / instruc&onal designers administrators / leaders / policymakers 40

41 How do analy&cs shape educa&on? By changing the system dynamics Faster feedback loops could enable more rapid adapta&on: of agents behaviour, and of learning resources and designs outcome intent researchers / educators / instruc&onal designers administrators / leaders / policymakers 41

42 How do analy&cs shape educa&on? Delega&on of authority to define goals, analy&cs, and meaning Distribution of power between educators, learners, leaders, community???

43 How do analy&cs shape educa&on? assessment epistemology the middle space of learning analytics pedagogy What epistemological assumptions are shaping the assessment regime, and hence the pedagogy? What questions are analytics used to help answer? Knight, S., Buckingham Shum, S. and Littleton, K. (In Press, 2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics. Open Access Eprint: 43

44 Example: epistemological assumptions Allows testing of problem-solving and analysis - sifting information "if you allow communication, discussions, searches and so on, you eliminate cheating because it's not cheating any more. That is the way we should think." Knight, S., Buckingham Shum, S. and Littleton, K. (In Press, 2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics. Open Access Eprint: 44

45 How do analy&cs shape educa&on? All of the above are encapsulated in any learning analy&cs deployment Figure from Doug Clow: hdp:// learning- analy&cs- cycle- closing- the- loop- effec&vely (slide 5) 45

46 How do analy&cs shape educa&on? What kinds of learners? What kinds of learning? What human +/or solware interven&ons / recommenda&ons? What data could be generated digitally from the use context? How is it cleaned? How to render the analy&cs, for whom, and will they understand them? What analy&cal tools could be used to find such paderns? Does your theory predict paderns signifying learning? 46

47 Conclusion: Analy&cs profoundly shape educa&on Ontologically Algorithmically Semio&cally Poli&cally Systemically Authority? 47

48 what kinds of learning are we op&mising the system for? 48

49 Learning analytics for this? The test of successful education is not the amount of knowledge that pupils take away from school, but their appetite to know and their capacity to learn. Sir Richard Livingstone,

50 Learning analytics for this? We re looking at the profiles of what it means to be effective in the 21 st century. [ ] Resilience will be the defining concept. When challenged and bent, you learn and bounce back stronger. Dispositions are now at least as important as Knowledge and Skills. They cannot be taught. They can only be cultivated. John Seely Brown US Dept. of Educ. conference (May 28, 2013) Dispositions clip: Whole talk:

51 Learning analytics for this? It s more than knowledge and skills. For the innovation economy, dispositions come into play: readiness to collaborate; attention to multiple perspectives; initiative; persistence; curiosity. Larry Rosenstock LearningREimagined project: Larry Rosenstock:

52 Learning analytics for this? In the growth mindset, people believe that their talents and abilities can be developed through passion, education, and persistence It s about a commitment to taking informed risks surrounding yourself with people who will challenge you to grow Carol Dweck Interview with Carol Dweck: Another interview:

53 Important work by Tony Bryk et al.: Drivers of Productive Persistence

54 Important work by Tony Bryk et al.: Drivers of Productive Persistence Note: a researchbased rationale for architecting a suite of analytics techniques

55 Bryk: sense of belonging a key predictor of remedial maths completion 55

56 Envisioning a wholistic university education (and analytics to match) 56

57 discourse learning analy5cs? 57

58 1 st International Workshop on Discourse-Centric Learning Analytics solaresearch.org/events/lak/lak13/dcla13 Beyond number / size / frequency of posts; hottest thread analytics that look beneath the surface, and quantify linguistic proxies for deeper learning

59 Discourse analytics on webinar textchat Can we spot the quality learning conversations in a 2.5 hr webinar? Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM

60 Discourse analytics on webinar textchat Sheffield, UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire, sunny 80 here! Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar See you! bye for now! bye, and thank you Bye all for now :28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 Average Exploratory Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM.

61 Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? 80 Not at the start and end of a webinar but if we zoom in on a peak :28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 Average Exploratory Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM.

62 Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar but if we zoom in on a peak Classified as exploratory talk :28 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 Averag (more substantive for learning) nonexploratory Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM.

63 Rhetorical discourse analytics OPEN QUESTION: little is known role has been elusive Current data is insufficient CONTRASTING IDEAS: unorthodox view resolves In contrast with previous hypotheses inconsistent with past findings... SURPRISE: We have recently observed... surprisingly We have identified... unusual The recent discovery... suggests intriguing roles De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012),

64 Rhetorical discourse analytics Human analyst 64

65 Rhetorical discourse analytics Human analyst Rhetorical parser De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012),

66 Rhetorical discourse analytics Glimpses of analytics capable of detecting higher order thinking. But humans will always read differently to machines Can we correlate this with academic writing, and can such analytics be used as formative feedback on drafts? Duygu Simsek s PhD: 66

67 Rhetorical discourse analytics CONTRAST SUMMARY & CONTRIBUTION Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: 67

68 disposi5onal learning analy5cs? 68

69 Dispositional Learning Analytics Workshop assessing-learning-dispositions-academic-mindsets 69

70 Assessing Learning Dispositions/Mindsets Observation informal and formal Future sweetspot... multiple lenses to provoke self-reflection Self-Diagnostic informal and formal Behavioural Analytics

71 Childrens informal self-assessment of managing distractions

72 Mindset Works mindsetworks.com Based on the educational research of Carol Dweck into growth mindsets (Stanford) 72

73 ELLI: Effective Lifelong Learning Inventory Ruth Deakin Crick

74 Quantifying learning dispositions agency; identity; motivation; responsibility A wholis&c visual, intended to build intrinsic mo&va&on, invi&ng stretch, providing a new language, provoking conversa&on that &es to the learner s iden&ty Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2 nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint:

75 Self-report through reflective blogging 9-10 yr old students Bushfield School, Wolverton, UK EnquiryBlogger Wordpress Multisite plugins 75

76 Masters level EnquiryBloggers Graduate School of Education, University of Bristol EnquiryBlogger: blogging for Learning Power & Authentic Enquiry 76

77 EnquiryBlogger teacher s dashboard direct navigation to learners blogs from the visual analytic 77

78 Systems leadership and learning: LearningEmergence.net 78

79 looking forward

80 Shifts in epistemic commitments? (Simon Knight, KMi PhD research) Does the way you search online reveal what you think counts as trustworthy knowledge? What is it to know when we search? Danish exams permit Net: Epistemic networks for epistemic commitments:

81 Shifts in epistemic commitments? (Simon Knight, KMi PhD research) Dimensions of Epistemic Belief Certainty Simplicity Source The degree to which knowledge is conceived as stable or changing, ranging from absolute, to tentative and evolving The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press).

82 What epistemic contributions are learners making in the community? Rebecca is playing the role of broker, connecting different peers contributions in meaningful ways We now have the basis for recommending that you engage with people NOT like you De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1 st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: 82

83 Dispositional profile from behavioural traces, to complement self-report? Questioning, arguing and search behaviours reveal intrinsic curiosity and epistemic commitments Social network patterns, teamwork effectiveness and initiation of relationships Tagging/sharing/ blogging/social patterns reveal how you see connections between ideas Behavioural and somatic traces associated with perseverance, grit, tenacity; overcoming panic/stress when stretched

84 The big shifts that analytics could bring Organisational Culture evidence-based decisions and org learning Academic Culture data-intensive learning sciences/ educ research Practitioner Culture evidence impact of learning designs; timely interventions C21 Qualities place these on a firm empirical evidence base 84

85 Critical zones for research+practice data-culture org. learning how do HEIs manage the embedding of real time analytics services? sensemaking meets computation creative intelligence + computational thinking pedagogical innovation how do learning analytics change student experience? educator data literacy how do staff learn to read and write analytics? 85

86 how to join in 86

87 The professional society 87

88 International Society Educational Data Mining" Every Year Learning Analytics Summer Institute 88

89 hdp:// 2/lasi2014/lasi- local 89

90 The Learning Analytics Conference March 16-20, 2015 (NY) 90

91 conclusion analy5cs will shape educa5on on mul5ple dimensions an analy5cs approach perpetuates an educa5onal worldview so let s ensure this is inten5onal not accidental...