Leveraging student evaluations for learning analytics

Transcription

1 Leveraging student evaluations for learning analytics

2 Outline Background Learning analytics Student evaluations in higher education Data requirements for using evaluations in learning analytics Example analyses incorporating evaluation data Future directions Leveraging student evaluations for learning analytics 2

3 Background UQ, Institute for Teaching and Learning Innovation (ITaLI) UQ is new to both learning analytics and online student evaluations. First pilot of online evaluations was conducted in Semester 1, Evaluations were fully online as of Semester 2, New Learning Analytics team commenced early 2015, and works together with the Evaluation Unit. Last 6 months: Examine current capabilities. Explore future opportunities. Leveraging student evaluations for learning analytics 3

4 Learning analytics The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs SoLAR, 2011 Improve student retention (Arnold & Pistilli, 2012). Examine student engagement and student performance as predictors of attrition (Aguiar et al., 2014). Identify popular course resources and usage patterns (Hecking et al., 2014). Inform course and program design (Ferguson et al., 2014). Detect on/off task behaviour (Baker, 2007) Leveraging student evaluations for learning analytics 4

5 Student evaluations in higher education can provide important evidence for assessing the quality of teaching, for supporting attempts to improve the quality of teaching and for informing prospective students about the quality of course units and programmes Richardson, 2005 At UQ: Student Evaluation of Course and Teaching (SECaT) Measure/monitor the quality of courses and teaching for quality assurance (Marsh, 2007) Research on teaching (Marsh, 2007) Source of information for students when selecting courses and/or teachers (Marsh, 2007) Measure of teaching quality for promotion and awards (Marsh, 2007) Source of feedback for improving courses and/or teaching (Marsh, 2007) Measure of student satisfaction Leveraging student evaluations for learning analytics 5

6 Student evaluations in higher education Online administration Most Australian universities have now adopted online student evaluations over paper-based methods. Improved accuracy Lower cost Quicker turn-around (reporting, etc.) Greater capacity for quantitative and qualitative analysis Debate continues regarding the use of online evaluations compared to paper-based evaluations, but generally the research shows: High response rates are still obtainable (at UQ ~ 42% average without incentives) Students prefer online evaluations Leveraging student evaluations for learning analytics 6

7 Data requirements for evaluations in learning analytics Confidential, online student evaluations Confidential evaluations allow for analysis down to the student level. Import student number as part of the data for scheduling evaluations, which can be matched to other sources of data. UQ s process protects students from being identified by anyone outside the Evaluation Unit (3 staff). Leveraging student evaluations for learning analytics 7

8 Data requirements for evaluations in learning analytics Data warehouse with multiple sources of data Readily available sources (in the Data Warehouse at UQ) include: SI-net (course enrolments, grades, timetabling, student demographics inc. ATSI, etc.) Electronic Course Profiles (teaching staff, assessment items, contact hours, etc.) Aurion (staff demographics, training/qualifications [limited], etc.) Student Evaluations (course and teaching evaluations) Queensland Tertiary Admissions Centre (student demographics inc. high school, OP, preferences, parents education, SES [postcode], etc.) UQ is the custodian of GO8 data (for comparison/benchmarking: enrolments, completions, demographics, teaching staff) Leveraging student evaluations for learning analytics 8

9 Data requirements for evaluations in learning analytics Data warehouse with multiple sources of data Other sources (not in the Data Warehouse yet!): LMS: Blackboard (Pyramid) Echo360 Library UQx Leveraging student evaluations for learning analytics 9

10 Example analyses incorporating evaluation data Pilot analysis Semester 1, 2015 One school only: 45 courses with evaluation data (>5 responses) Average of 171 responses per course 7,698 unique students 2,704 students participated in evaluations Leveraging student evaluations for learning analytics 10

11 Example analyses incorporating evaluation data Participation analysis: Pass/Fail Leveraging student evaluations for learning analytics 11

12 Example analyses incorporating evaluation data Participation analysis: Grade and GPA Grade GPA Completed SECaT Did not complete SECaT Leveraging student evaluations for learning analytics 12

13 Age (years) Example analyses incorporating evaluation data Participation analysis: Age Completed SECaT Did not complete SECaT Leveraging student evaluations for learning analytics 13

14 Example analyses incorporating evaluation data Participation analysis: Gender Leveraging student evaluations for learning analytics 14

15 SECaT Q8 Example analyses incorporating evaluation data Correlation between academic performance (pass/fail) and student evaluations Pass 3.57 Fail Leveraging student evaluations for learning analytics 15

16 Example analyses incorporating evaluation data Correlation between academic performance (grade) and student evaluations SECaT Grade r =.176 p <.001 Not to scale! Leveraging student evaluations for learning analytics 16

17 Example analyses incorporating evaluation data Predicting academic performance based on student evaluations and demographics SECaT Grade Gender (F) Age (older) Not to scale! Leveraging student evaluations for learning analytics 17

18 Future directions based on broader data Moving from myth-busting to wider implementation Adding predictions of academic performance (based on demographics and evaluations) to course reports. Apply analyses to full set of UQ evaluations data. Incorporate qualitative analysis into course reports. Refine communication strategies to target the groups of students who aren t engaged with student evaluations and courses. Refine support strategies to target lower performing students and improve course outcomes. Presentation Title 18

19 References Aguiar, E., Ambrose, G. A., Chawla, N. V., Goodrich, V., & Brockman, J. (2014). Engagement vs performance: Using electronic portfolios to predict first semester engineering student persistence. Journal of Learning Analytics, 1(3), Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. Paper presented at the 2 nd International Conference on Learning Analytics and Knowledge, Vancouver, British Colombia, Canada. Baker, R. S. J. D. (2007, April). Modeling and understanding students off-task behavior in intelligent tutoring systems. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp ). ACM. Ferguson, R., Macfadyen, L. P., Clow, D., Tynan, B., Alexander, S., & Dawson, S. (2014). Setting learning analytics in context: Overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), Hecking, T., Ziebarth, S., & Hoppe, H. U. (2014). Analysis of dynamic resource access patterns in online courses. Journal of Learning Analytics, 1(3), Marsh, H. W. (2007). Students evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In The scholarship of teaching and learning in higher education: An evidence-based perspective (pp ). Springer, Netherlands. Richardson, J. T. E. (2005). Instruments for obtaining student feedback: A review of the literature. Assessment and Evaluation in Higher Education, 30(4), SoLAR (2011). Open Learning Analytics: An Integrated & Modularized Platform. White Paper. Society for Learning Analytics Research. Retrieved from Leveraging student evaluations for learning analytics 19

20 thank you!