Running head: LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!1. Learning Analytics as Assessment for Learning. Lisa Rubini-LaForest

Transcription

1 Running head: LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!1!!!!! Learning Analytics as Assessment for Learning Lisa Rubini-LaForest University of Calgary

2 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!2 Abstract Learning analytics is a emerging field in education research. In this paper, I discuss using learning analytics specifically as a means of completing assessments for learning in the K-12 education sphere. After discussion of both assessment and learning analytics in general, specific descriptions of how the objectives of learning analytics match the objectives of assessment for learning are provided. Examples of current projects using learning analytics to improve student learning are given as appropriate. The paper ends with a discussion of the challenges and needs associated with using learning analytics as assessment for learning and applications to my own personal practice.

3 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!3 Learning Analytics as Assessment for Learning Learning Analytics (LA) is an emerging technique for interpreting learner data into actionable intelligence (Clow, 2012, p. 134) which is then used to improve learning. LA use techniques taken from a variety of research fields and then specialize those techniques into the educational sphere (Chatti et al, 2012). Learning analytics were described by Johnson et al (2013) in the 2013 Horizon report as a technology that is currently at the mid-term horizon (p. 4) of adoption with 20% usage expected within 2-3 years. From this initial understanding of learning analytics, it became clear to me that assessment for learning was a natural application of LA. Wide spread adoption of using LA would specifically address one of the significant challenges listed in the 2013 Horizon report by allowing teachers to use digital media for increased formative assessment (Johnson et al, 2013). Personally, I chose to examine this topic after finding the work of manually tracking student taking my elearning course tedious, time-consuming, and repetitive. As I believe that my manually tracking and directed interventions with students lead to increased success of my students, I have been convinced that learning analytics would be a useful assessment tool for both teacher-led and student-led assessment. This paper looks at how LA is a means of assessment and how it could be used to provide assessment for learning to K-12 teachers and learners. To begin, a definition of assessment will be provided and then discussed to apply the context which is used in this paper.. After LA is defined, this paper looks at using LA to provide assessment for learning. Examples of LA being used in the K-12 educational realm are described and the paper ends with a description of future concerns and needs for assessment using LA.

4 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!4 Assessment Assessment is defined by the Ontario Ministry of Education (2010) as the process of gathering information that accurately reflects how well a student is achieving the curriculum expectations in a subject or course with a primary purpose... to improve student learning (p. 28). I have chosen the definition here from the Ontario context as it best matches my teaching practice. The Ontario Ministry of Education (2010), follows the lead of Harlan (2006, as cited in Ontario Ministry of Education 2010) in replacing the words formative, diagnostic, and summative with the terms assessment as learning, assessment for learning, assessment of learning (see Figure 4.1 on p. 31). This change acknowledges that it is not the collection of information but rather the actions that the information initiates which determine that type of assessment which has occurred. Assessment for learning contains both diagnostic and formative assessment. Formative assessment uses information to improve learning during a learning unit. Diagnostic assessment is given before instruction has begun and determines what knowledge a learner already has about a given topic (Ontario Ministry of Education, 2010). Both formative and diagnostic assessment can use feedback given by the learner, peers, or a teacher but the actions caused by the assessment are generated by the teacher or learner (Black and William, 2009). Evaluation is the process of judging the quality of student learning on the basis of established performance standards... and assigning a value to represent that quality. (Ontario Ministry of Education, p. 38). Evaluation is not considered in this paper as it is not a form of assessment for learning.

5 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!5 Learning Analytics Learning Analytics are defined by Chatti et al (2012) as methods for analyzing and detecting patterns within data collected from educational settings and [leveraging] those methods to support the learning experience (p. 5). Ferguson (2012) notes that generalized definitions of LA actually encompass all educational research and adds the techniques of LA use machinereadable data and the techniques of LA must be able to handle big data sets. LA is a new field which uses or specializes methods from many different fields: academic analytics, action research, educational data mining, recommender systems, and personal adaptive learning. The techniques used in these basis fields include machine learning, descriptive statistical analysis, and large data management techniques amongst others (Chatti et al, 2012). LA can be used to meet many objectives. Chatti et al (2012) provide a list of 7 objectives which encompass the purposes of most LA: monitoring and analysis of learners; prediction of success and intervention to increase success; tutoring and mentoring learners; assessment and feedback about the learning process; adaptation of learning materials to a students readiness, interest and learning styles; personalization and recommendations to support independent learning; and reflection after learning by both learner and instructor. LA can be used at a variety of stages in the learning process. Students, teachers, administrators, and policy makers can use learning analytics to inform their practices (Clow, 2012). Clow (2012) notes that the it is important that LA be used to create actionable intelligence (p. 134). Clow s cycle of identifying learners; collecting data; calculating metrics; and determining interventions; and applying those interventions back onto learners ensures that LAs are used to improve the learning process and not merely to study it.

6 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!6 Learning Analytics and Assessment Comparing the definitions of assessment and LA of this paper, the similarities are clear. Both assessment and LA are innately focused on turning student data into actionable information which is used to improve student learning. LA could be used as assessment for learning. Following the lead of Clow (2012), it appears that using LA for assessment is in fact a subset of assessment for learning. The change is what data is being collected. Whereas traditionally a teacher provided assessment for learning using paper assessment, informal, formal observation, and checklists (Black and William, 2009); LA allows formative assessment metrics to be created using machine-readable data and data sets large enough to need to be handled by a computer (Ferguson, 2012). The overall goal of this assessment remains the same: to improve student learning. Learning Analytics as Assessment for Learning Assessment for learning occurs during instruction with the ultimate goal of allowing the teacher or student to alter instruction, goals, or teaching methods to improve student learning and achievement (Black and William, 2009). Two key aspects of assessment for learning as described by Black and William (2009) relate strongly to the use of LA: providing feedback that moves learners forward and activating students as the owners of their own learning (p. 8). Providing feedback that moves learners forward Of the 7 objectives of LA as provided by Chatti and al (2012), two meet the objective of providing feedback to students to increase their learning: prediction and intervention; and adaptation. Prediction and Intervention. In prediction and intervention, data collected from a

7 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!7 learner is used given existing models to predict whether or not a student is likely to be successful in the current learning unit or course. This analysis requires access to similar data provided by learners in a similar learning context so that the trends in who is successful and who is high-risk can be determined and then used to diagnose the current group of learners. After students are identified in the prediction stage, a teacher or course created intervention is provided to the student to increase their chances of success. (Chatti et al, 2012) There are two main methods of completing such analysis. The first is to complete a statistical analysis of the data, using varied sources such as LMS logs, social network analysis, or contextual analysis to determine a learner profile of a successful learner and then match current students to this pre-existing model to predict success (Chatti et al, 2012). The second way would be to create a machine learning model which is trained by existing data. Once trained, the model is used to predict learner success. A key addition to the machine learning model is that the model is trained further using each set of learner data after the learning unit. Using this technique, the machine learning model becomes more accurate the more students it is used with. This machine learning model would be a stronger application of LA since it allows the predictive model to adjust to changes in student needs or characteristics (Russell & Norvig, 2011). Prediction and intervention is assessment for learning as it allows the course instructors to determine which learners are considered at risk of not succeeding in the course using set metrics. This is very similar to how students have been identified as high-risk at the schools I have worked at during my career. However, using larger data sets allows LA to create a finer model and, hopefully, for predictions to be more accurate. Once prediction has occurred, just as in traditional assessment, the loop must be closed (Clow, 2012) and additional supports such as scaffolds, additional learning resources, or one-to-one assistance must be provided as

8 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!8 interventions. One example of using prediction and intervention is the S3 model created by Essa and Ayad (2012). This model used numeric variables taken from LMS logs such as number of discussion messages posted, total number of mail messages sent, and total number of assessments completed. (p. 159) Using these and other variables, the S3 model creates a variety of risk measurements and displays these results to the instructor using scatterplots and win/lose visualizations. Chatti et al (2012) note that visualizations are important aspects of the LA cycle. If LA is to be adopted by mainstream educators, the visualizations LA creates must be understandable and usable by people who are not technical in training. Using well known visualizations such as scatterplots and win/lose charts taps into the instructor s statistical understanding and allows the instructor to see how well each learner is doing in the course. From there it is up to the instructor to intervene in the student s best interest. Another example of prediction and intervention is the Purdue Signals project. In this project, LA is used to determine an overall risk analysis for each student based on LMS variables (Arnold, 2010). This system uses a black-box and does not allow the student or instructor to see which variables the students appears to be high risk within, instead providing a simple signal to the student and instructor (Essa and Ayad, 2013). If students see a green light, they know that they are not exhibiting risk behaviours within the LMS; if students see an amber light, they know they are exhibiting borderline behaviours within the LMS; and if the students see a red light they know that the model has predicted them to be at high risk of not succeeding in the course. At this point, both the students and teaching staff are able to intervene in the students best interests. (Arnold, 2010).

9 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!9 Adaptation. Adaptation controls the flow of a learner s tasks by adapting to a learner s needs and successes dynamically as the user learns (Chatti et al, 2012). Such a system can be simply implemented using the release tools found in most major LMS. In an LMS system you are able to release activities for a learner to complete based off of the achievement of the learner on a previous assignment. A more advanced version would be the use of an Intelligent Tutoring System (ITS) or recommendation system which changes its suggestion for each learner based on a large number of user variables provided by LA. (Chatti et al, 2012). Adaptation is another example of assessment for learning as it allow just-in-time adaptation of learning resources based off of performance. The goal of the adaptation continues to be the growth and success of the learner. An example of adaptation is the ITS ASSISTment which is used in Massachusetts middle schools. This system allows students to practice mathematics techniques with adaptive scaffolds. In other words, depending on how a student does in any one part of a question affects what assistance or questions the student will receive next (Razzaq et al, 2005). Feng and Heffernan (2010) further demonstrate that the assessment and preparation given by the system improved test-scores better than online paper partially because the system provided better formative assessment and assistance to students. In ASSISTment, the ITS was able to adapt learning based on richer metrics about student work than would be available on a homework practice-test: specifically, the number of time a student requested help while completing the test, and response time per question. Activating Students as the Owners of their own Learning Of the 7 objectives of LA as provided by Chatti and al (2012), three meet the objective of activating students as the owners of their own learning: prediction and intervention,

10 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!10 personalization and recommendation, and reflection. Prediction and Intervention. The techniques described in the previous section all apply here as well. The difference is how the predictive information is used. When the predictive information is used by the student to acknowledge weaknesses in their learning method and take corrective action, this objective transforms into a method of increasing learner ownership in their learning. An example of this was Purdue s Signals project which allowed students to view their risk measure and provided key resources for taking corrective action (Arnold, 2010). Personalization and recommendation. Personalization and recommendation are new concepts in institutional learning as they use LA to allow students to choose their own learning path using a knowledge-pull learning model. In this objective, LA is used to determine the interests, needs, and preferences of the student and then personalize the learning unit using the student as the key accessor and director of his or her own learning (Chatti et al, 2012). These systems are reminiscent of the recommendation and personalization seen in commercial ventures such as Google, Amazon, or other marketing/advertising companies. Personalization and recommendation is assessment for learning as it allows learners to see how their actions during learning affect the offerings given to the them by the program. Chatti et al (2012) recommend that this objective be implemented within a personal learning environment (PLE). Since PLEs are an emergent direction in educational theory and practice, there were not scholarly examples found of this work. Reflection. Chatti et al (2012) describe reflection as comparing data to improve continuing learning. While most reflection occurs at the end of a learning unit, reflection can occur mid-unit when learners are provided good visualizations which allow them to pause and consider their current learning practices.

11 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!11 An example of using reflection is Tempelaar et al s (2013) use of learning analytics within test-directed learning systems. Based off their work in the test-directed system, students were provided with LA visualizations which allowed students to view their current progress in the course, determine their current level of mastery, and compare their progress to their peers. This simple use of LA allowed students who were low-achieving to realize they needed to spend more time in the test-driven learning environment to be successful. Use of Learning Analytics as Assessment of Learning in K-12 Education There is little scholarly work that has so far implemented the use of LA for the use of assessment for learning in the K-12 educational realm. Currently, most of the work on the topic seems to be aimed at what Chatti et al (2012) called the objective of monitoring and analysis. According to Clow (2012) these activities do not even complete the LA cycle as they do not create actionable intelligence which is implemented by the learners or institution. For instance, the while the ASSESSment program used by school districts in Massachusetts allows for formative feedback in the form of adapted further questioning, there is minimal reporting out of the system to school teachers in a timely manner such that the feedback given could be used to improve student learning in the short term (Feng and Heffernan, 2010). Indeed, Feng and Heffernan s (2010) main goal in their study was to determine if using the ITS was an effective at improving standardized test scores instead of looking at how to make the ITS more predictive and successful as a teaching tool. Similarly, the use of Betty Brain by Segedy et al (2013) focused on analysis instead of assessment. In this model, students were asked to teach Betty Brain by creating content flowcharts. The analysis of the LA was used to determine how successfully the learners obtained knowledge and were able to complete the activity. This is particularly disappointing since the use

12 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!12 of Betty Brain was largely unsuccessful and if Segedy et al had used an adaptive objective with the LA they would have been able to change the learning scheme to allow better student success. Lastly, Monroy et al (2013) s project one again only focused on monitoring and analysis of learning. STEMscopes is a tool used by K-12 students to supplement and add to their science curriculum. By analyzing LA of the system, Monroy and his associates were able to learn about how students and teachers use the online resources within a traditional classroom. In the further work section, Monroy acknowledges that further work in the system could apply the LA to student assessment creating a richer tools. It should be noted however, that many of the tools used at the university level could be used in e-intensive or e-focused classrooms at the K-12 learning level. Particularly the S3 project developed by Essa & Ayad (2012) or Purdue s Signals project (Arnold, 2010) would be excellently placed into K-12 learning materials where students frequently use the LMS to guide their learning. Needs and Challenges in using LA as Assessment for Learning Privacy and ethics are a major concern of people considering the use of LA. Arnold (2010) notes that while they have not had any student complain about the use and analysis of their data, it does not alleviate the organization from the ethical concerns created from it. Similarly, both Ferguson (2012) and Chatti et al (2012) list ethical and privacy concerns as key challenges in the adoption of LA. As school districts adopt the use of big data to improve student learning it is increasingly important that data security measures such as encryption and anonymization, account access restrictions, network security as well as physical security techniques such as clean desk policies, locked access to computers and physical storage, and monitoring of access occur. As well, it is important that all stakeholders understand what data is

13 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!13 and is not being collected so that parents, teachers, and learners can all have time to understand, consent or object to the type of data being collected. Another major concern is staff professional development. Chatti and al (2012) acknowledge that once widely adopted the vast majority of LA users and implementers will not be experts in LA or even technology and the tools created must be able to be easily used and understood. Furthermore, it is important that teachers and learners be well aware of what the metrics are telling them and specifically what they are not telling them. The amount of general statistical understanding and critical analysis skills needed to appropriately use and understand LA is significant and it will require training and learning on the part of all stakeholders if using LA for assessment is to be successful. The last need with using LA as assessment for learning is the need to more fully integrate discussion and social learning into the LA metrics. While there are many project currently attempting to determine computational solutions to this practice (see references of Chatti et al, 2012; and Ferguson, 2012), none of these projects are nearly robust enough yet to allow for use within the K-12 educational sphere. Conclusions & Implications to my Personal Practice As a teacher who has the privilege to work within a computer lab at all times, I see great potential in using LA within my classroom to increase student learning. If I could predict and intervene with students more quickly and accurately it would save students frustration and myself time as students would receive the interventions they need when they first start demonstrating symptoms of needing help rather than at the end of a learning unit. Ontario has a large push for personalized and adaptive education through the use of differentiated instruction. Using LA for adaptation, personalization, and recommendation, I could

14 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!14 provide my students with choices that would be serve their readiness, preparation, and needs while still allowing them choice based on their preferences and interests. Goal setting is a topic I have stressed in my classroom throughout the last school year. Using LA for reflection, I could continue to help my students make wise revisions to their goals by allowing them to monitor their learning progress and consider their learning habits. Student would be able to make decision using factual data rather than depending on their recollection of past learning and tasks. Lastly, I could see LA automating my daily what you completed s that I sent daily throughout my e-intensive teaching experience. I could be given a customizable report of daily activity. Being customizable is important because that would allow me to still put my personal touch on the and I know my students really appreciated that each was from me directly and not an automated . However, the collection, sorting and visualization of attendance data, assignment submissions, quiz completion, and discussion forum activity is something that would be far more quickly completed by a program. Unfortunately, until major LMS makers create modules to provide LA based assessment for learning, I doubt that I will be able to make use of the techniques I have explored in this paper in any manner other than manual creation of LA when time permits. While my computer science background would allow me create my own tools and widgets based off of data retrieved from computer logs, surveys, and LMS variables, I simply do not have time to be both a developer and a teacher. As well, it can be difficult to obtain access to all the data which is used in good LA analysis. LA have great potential for use in the K-12 online classroom and hopefully Johnson et al were correct in their Horizon report (2013) that there will be mainstream adoption of the

15 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!15 technology within 2-3 years. This adoption would lead to greater research on the topic and would refine its practices to allow teachers to use the vast amounts of data created by learners online to increase success and effective learning.!

16 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!16 References Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1), n1. Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (formerly: Journal of Personnel Evaluation in Education), 21(1), Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2012). A reference model for learning analytics. International Journal of Technology Enhanced Learning, 4(5), Clow, D. (2012, April). The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp ). ACM. Essa, A., & Ayad, H. (2012, April). Student success system: risk analytics and data visualization using ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp ). ACM. Feng, M., & Heffernan, N. (2010, June). Can we get better assessment from a tutoring system compared to traditional paper testing? can we have our cake (better assessment) and eat it too (student learning during the test)?. In Proceedings of the 10th international conference on Intelligent Tutoring Systems-Volume Part II (pp ). Springer-Verlag. Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5), Johnson, L., Smith, R., Levine, A., and Haywood, K., (2013) Horizon Report: K-12 Edition. Austin, Texas: The New Media Consortium. Monroy, C., Rangel, V. S., & Whitaker, R. (2013, April). STEMscopes: contextualizing learning analytics in a K-12 science curriculum. In Proceedings of the Third International

17 LEARNING ANALYTICS, ASSESSMENT FOR LEARNING!17 Conference on Learning Analytics and Knowledge (pp ). ACM. Norvig, P., & Russell, S. (2011). Artificial intelligence: a modern approach. Pearson Higher Ed. Ontario Ministry of Education. (2010). Growing Success assessment, evaluation and reporting: Improving student learning. Toronto: Queen s Printer for Ontario. Razzaq, L. M., Feng, M., Nuzzo-Jones, G., Heffernan, N. T., Koedinger, K. R., Junker, B.,... & Rasmussen, K. P. (2005, May). Blending Assessment and Instructional Assisting. In AIED (pp ). Tempelaar, D. T., Heck, A., Cuypers, H., van der Kooij, H., & van de Vrie, E. (2013, April). Formative assessment and learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp ). ACM. Segedy, J. R., Loretz, K. M., & Biswas, G. (2013, April). Model-driven assessment of learners in open-ended learning environments. In Proceedings of the Third International Conference! on Learning Analytics and Knowledge (pp ). ACM.