Application of qualitative and quantitative content analysis to course evaluations: Technical challenges and opportunities

Transcription

1 Application of qualitative and quantitative content analysis to course evaluations: Technical challenges and opportunities Normand Péladeau President Provalis Research Corp. Sameer Saab CEO explorance

2 Our Products 1989 Statistical Analysis & Bootstrapping 2004 Qualitative Analysis & Mixed Methods 1998 Content Analysis & Text Mining

3 Founded in 2003 Developer of Blue Software LEM solution Primary Focus: Education, Learning and Development 200 clients globally

4 About Blue Learning Experience Management (LEM) software, fully integrated, mainly targeted at assisting organizations in implementation of a culture of continuous improvement. Blue/360 PEER EVALUATIONS 360 FEEDBACK REVIEWS UPWARD FEEDBACK Blue/Evaluations COURSE EVALUATIONS SERVICES EVALUATIONS DEPARTMENT EVALUATIONS Blue/Surveys EMPLOYEE/CUSTOMER SURVEYS ONLINE ELECTIONS EXIT INTERVIEWS Blue/Tests SELF TESTS PSYCHOMETRIC TESTS (Q4/2014 ) ONLINE EXAMS (Q4/2014 )

5 Blue Technology Advantage

6 Create, build & distribute custom course evaluation

7 Detailed analysis

8 Create, build & distribute custom course evaluation

9 Blue Text Analytics System Diagram Input Text comments from online course evaluation forms, compiled with Blue Blue Course Evaluations Provalis Text Analytics Engine (SDK) Dictionary for Post-Secondary Education Output Secure reports compiled for instructor, department, division, or institution

10 Chart & Table of Theme Frequency

11 Challenge #1 Question Formulation What are the strengths and weaknesses of this seminar? Please discuss the reasons for your overall rating of the instructor. What was most helpful about instructor's teaching methods? What aspects of this course are in most need of improvement? What is the best aspect of RMIT? What can this instructor do to improve teaching effectiveness in the classroom? What are the strengths of this course? How could the instructor improve the course? This unit could be changed in the following ways to improve my learning: What are the strengths and weaknesses of the instructor? Describe the most helpful/effective aspect(s) of the course. Would you recommend this course to other students? Explain. Would you recommend this instructor to other students? Explain How do you think the instructor could improve this course?

12 Challenge #2 Regionalism English Countries: Australia; Canada; Dubai; New Zealand; United Kingdom; United States. Courses: United States Courses, Subjects, Lectures, Modules Australia Subjects, Units, Courses New Zealand Papers, Sections Teacher: Canada Instructors, professors United States Faculty, instructors Australia Teachers, professors

13 Challenge #3 Quantity Corpus of 1,8 million comments 35 millions words (tokens) word forms (types)

14 Challenge #4 Misspellings word forms (types): unknown words 75 % misspellings ( 35,000) o 21 % proper names o (Products & People) 4% acronyms o

15 Challenges Misspellings 61 ways to be Enthusiastic

16 Laziness Necessity is the mother of invention

17 Challenges Speed Test Data : 904,337 cases / 10,5 M. words v 6.1 v 7.0 Phrase Extraction : 45,071 phrases Keyword-In-Context Helpful (19,205) Weather (100) 398,6 sec. 349,1 sec. 337,9 sec. 119 Sec. 9,2 sec. 0.3 sec.

18 KWIC 6.1

19 KWIC 7.0

20 Challenges Misspellings 61 ways to be Enthusiastic

21 Challenges Misspellings in v6.1

22 Challenges Misspellings in v6.1 Fuzzy and phonetic string comparison algorithms: Damerau-Levenshtein Koelner Phonetik SoundEx Metaphone Double-Metaphone NGram Dice Jaro-Winkler Needleman-Wunch Smith-Waterman-Gotoh Monge-Elkan

23 Challenges Misspellings in v6.1

24 Challenges Misspellings in v6.1

25 Challenges Misspellings in v6.1

26 Challenges Misspellings in v7.0

27 Challenges Named Entity Extraction

28 Creation of a dictionary Current course evaluation dictionary 10,323 words (including > 4000 misspelling) 151 categories and subcategories 3 global types of categories 1. Attributes 2. Objects 3. Alerts

29 Creation of a dictionary

30 Creation of a dictionary

31 Creation of a dictionary

32 Creation of a dictionary

33 Blue Text Analytics System Diagram Input Text comments from online course evaluation forms, compiled with Blue Blue Course Evaluations Provalis Text Analytics Engine Dictionary for Post-Secondary Education Gateway to ProSuite tools QDA Miner SimStat Custom Dictionary B Output Custom Dictionary C Secure reports compiled for instructor, department, division, or institution WordStat In-depth analysis with ProSuite, optional desktop software from Provalis

34 Blue Text Analytics Deeper Research with ProSuite Online Course Evaluations from Students Suggestions Freeform Comments Opinions ProSuite desktop tools for in-depth analysis (optional) Blue Text Analytics Gateway to ProSuite

35 What should be driving innovation in CAQDAS?