Depok, 19 November 2015

Transcription

1 Depok, 19 November 2015 MIXED-METHODS IN BEHAVIORAL RESEARCH: ONLINE LEARNING AND HUMAN-COMPUTER INTERACTION Harry Budi Santoso, PhD Faculty of Computer Science, Universitas Indonesia

2 Professional Contribution Chief Editor of Jurnal Sistem Informasi (JSI) Program Chair of ICACSIS 2014 & 2015 Program Chair of Developing Online Education Seminar & Workshop 2014 Organizing Committee of the 1 st and 2 nd International Conference on Human- Computer Interaction and User Experience Reviewer of Asia Pacific CHI UX 2015 (@ Symposium) Reviewer of American Society for Engineering Education Annual Conference Reviewer of ASEE/IEEE Frontiers in Education Annual Conference Reviewer of ICACSIS Annual Conference Reviewer of ICAICTA Invited as a Journal of Educators Online Invited as a International Review of Research in Open and Distance Learning

3 Research Interests Online Learning/Computer Assisted-Instruction Human-Computer Interaction/User Experience Metacognition/Self-Regulated Learning Engineering/Computer Science Education

4 Journal Writing Experience Computers in the Schools (resubmitted, 2015) The International Review of Research in Open and Distance Learning (published, 2015) Journal of Pre-College Engineering Education Research (published, 2014) Journal of Educators Online (published, 2014; accepted, 2015) MERLOT Journal of Online Learning and Teaching (published, 2014) International Education Studies (published, 2013) International Journal of Engineering Education (published, 2013) Design and Technology Education: An International Journal (published, 2013) Journal of STEM Education: Innovations and Research (published, 2013) Journal of Educational Technology & Society (published, 2012)

5 International Conferences The 23rd International Conference on Computers in Education 2015, Hangzhou, China The 3rd International Conference on User Science and Engineering 2014, Shah Alam, Malaysia The 2013 ASEE/IEEE Frontiers in Education conference, Oklahoma City, Oklahoma, USA. The 2013 American Society of Engineering Education (ASEE) annual conference, Atlanta, Georgia, USA The 2012 ASEE/IEEE Frontiers in Education annual conference, Seattle, Washington, USA The 2011 IEEE/ASEE Frontiers in Education annual conference, Rapid City, South Dakota, USA

6 Why do we need to publish?

7 Why do we need to publish? austinkleon.com

8 The steps to get our article published HIGH QUALITY RESEARCH TO HIGH IMPACT JOURNAL PUBLICATION Conducting Research Submitting the Research Paper Before Submitting to Journal Publication After Submitting to Journal Publication

9 Interdisciplinary Nature of Online Learning & HCI Research Computer Science Education Psychology Sociology Anthropology Engineering

10 Online Learning Framework

11 The Content of Human-Computer Interaction (SIGCHI.ORG)

12 Types of Behavioral Research (Rosenthal & Rosnow, 2008) Descriptive Investigations Keyword: Describe a situation or condition Typical Methods: Relational Investigations Keyword: Identify relations between (two or more variables) Typical Methods: Experimental Investigations Keyword: Identify causes of a situation or condition Typical Methods:

13 Defining Mixed Methods Involved integrating quantitative and qualitative approaches to generating new qualitative approaches to generating new knowledge and can involve either concurrent or sequential use of these two classes of methods to follow a line of inquiry. Stange K et al (2006). Integrating quantitative and qualitative data collection and analysis in a single study or a program of enquiry. Creswell et al 2003.

14 What is Mixed-Method Research? focusing on research questions that call for real-life contextual understandings, multi-level perspectives, and cultural influences; employing rigorous quantitative research and rigorous qualitative research; utilizing multiple methods (e.g., intervention trials and in-depth interviews); (National Institutes of Health, Office of Behavioral and Social Sciences Research)

15 Download: well_a%20framework%20for%20design.pdf

16

17

18

19 RESEARCH EXAMPLE Computer Self-Efficacy, Cognitive Actions, and Metacognitive Strategies of High School Students While Engaged in Interactive Learning Modules Harry B. Santoso

20 Background Educators and policy makers are engaged in an effort to improve the teaching of STEM subjects in the United States. Introductory concepts of computer science, for example programming, are difficult to learn (Denning, 2004) and unattractive/unappealing (Jepsen and Perl, 2002). Research suggests that computer applications can be used to increase learning and keep learners interested. Initiatives have been made to promote the use of ICT in education: The NSF Cyberlearning: Transforming Education program has objective to better understand how people learn with technology and how technology can be used productively to help people learn (NSF, 2011, p. 1).

21 Rationales Self-Regulated Learning (SRL) as a major construct in educational research learning process Despite the growing interest in SRL research, few studies have investigated the degree to which the students are aware of their thinking process while working on interactive learning module (ILM) in high school level. A very limited instrument is available to investigate SRL skills while learning using ILM. Learning using computer applications requires SRL skills higher than learning with instructor in classroom (e.g., Chang, 2005). Suggestions are needed to improve Interactive Learning Modules

22 Computer-Self-Efficacy, Cognitive Actions, and Metacognitive Strategies in SRL Framework Self-regulated learners are metacognitively, motivationally, and behaviorally active participants in their own learning process (Zimmerman, 1989). SRL is a complex, situated, dynamic process involving individuals learning in context (Butler & Cartier, 2005) Self-Regulated Learning in Context Motivation Metacognition Computer Self-Efficacy Behavior Cognitive Actions Planning Strategies Monitoring Strategies Regulating Strategies

23 Research Questions How is students computer self-efficacy (CSE) related to cognitive and metacognitive strategies while using interactive learning modules (ILM)? Sub-question: What is the relative importance of CSE with regards to its contribution toward students cognitive actions and metacognitive strategies while using ILM? How do students plan and monitor their cognitive actions, and regulate their monitoring strategies during learning with ILM? Sub-question: How do high and low CSE students plan and monitor their cognitive actions, and regulate their monitoring strategies during learning with ILM?

24 The Study Participants and Context School Selection: Participant Selection: School Class Logan High School Programming 1A and Math 1 InTech Collegiate High School Physics 100 students from both schools completed all activities in this study. Three modules for each class were selected to be used by considering the relevance of the modules to this study.

25 Features of the Modules Features Boolean Logic Minimum Spanning Tree Modeling Using Graphs Readings Instructions Exercises Level of difficulties

26 The Use of Multimethods This research will use three data collection methods: Online survey instruments Demographic questionnaire Computer Self-Efficacy questionnaire Self-Regulated Computer-Based Learning questionnaire Interactive Learning Module screen captured videos Interviews

27 Computer Self-Efficacy Questionnaire The survey will be used to understand students judgment of capabilities to use computers in different situations (Compeau, Higgins, and Huff, 1999; Marakas, Yi, & Johnson, 1998) This questionnaire was adapted from the work of Durndell, Haag, & Laithwaite (2000). The CSE questionnaire responses ranged from 1 to 5 (i.e., 1 = not at all true of me and 5 = very true of me). Scale Cronbach s Alpha (Original) Cronbach s Alpha (The Study) Beginning Skills Advanced Skills File and Software Skills

28 Self-Regulated Computer-Based Learning Questionnaire The survey was developed to capture students perception of their cognitive actions and metacognitive strategies while learning using interactive learning modules. This questionnaire was adapted from the work of Lawanto (2011) based on Butler and Cartier s SRL theoretical model (Butler & Cartier, 2005; Cartier & Butler, 2004). Measurement scales of EDQ items ranged from 1 to 4 (i.e., 1 = almost never, 2 = sometimes, 3 = often, and 4 = almost always). Scale Cronbach s Alpha (The Study) Planning Strategies.694 Cognitive Actions.812 Monitoring Strategies.878 Regulating Strategies.814

29 Data Collection Procedures Completing Demographic & CSE online surveys Learning with the modules Recorded using screen-capture software Completing SRCBL online survey Selected students were interviewed

30 Data Analysis Answering Research Question 1 and Sub-question Research question How is students computer self-efficacy (CSE) related to cognitive and metacognitive strategies while using ILM? Procedure Method Purpose The mean values of CSE and SRCBL items will be calculated using descriptive statistics and graphical views Correlation tests will be conducted by using Pearson tests Profiling of CSE, cognitive, and metacognitive strategies. To measure the relationships between: (1) CSE and cognitive actions, and (2) CSE and metacognitive strategies Multiple linear regression tests To measure the relative importance of CSE with regards to its contribution toward: (1) cognitive actions, and (2) metacognitive strategies

31 Data Analysis Answering Research Question 2 and Sub-question Research question How do students plan and monitor their cognitive actions, and regulate their monitoring strategies during learning with ILM? Procedure Method Purpose Repeated measures Cluster analysis Screen-captured video analysis Interview analysis To measure significant differences between: (1) planning and cognitive actions, (2) monitoring and cognitive actions, and (3) monitoring and regulating strategies. To determine which screen-captured videos will be analyzed and to select students need to be interviewed. To explain findings from questionnaire analysis about how Cognitive Actions were planned and monitored, and how Monitoring Strategies were regulated. To explain findings from questionnaire and screencaptured video analysis.

32 Analyzing Screen-Captured Videos 1 Students interactions with ILM were captured & transcribed List of events were transformed into meaningful sequence of events Graph that visualize dynamicity of strategy changes and duration for each strategy Coding process: Cognitive actions, Planning, Monitoring, and Regulating Strategies Frequency of strategy changes Duration of strategies

33 Findings of the Study and Discussion

34 Demographic Information One-hundred students (77 males and 23 females) completed all activities in this study during the spring 2013 semester. About 62% of the participants had a GPA 3.00 or higher. About 66% participants were considering majoring in a field of engineering, technology, or computer science.

35 Data Homogeneity Data from CSE and SRCBL questionnaires were used in the analysis to investigate whether differences existed among the participants. The findings revealed that there was no significant difference between Logan and InTech Collegiate High Schools insofar as their computer self-efficacy, cognitive actions, planning, monitoring, regulating, and overall metacognitive strategies. In summary, these findings suggested that the data collected from both schools were homogeneous.

36 Addressing Research Question #1 How is students CSE related to cognitive actions and metacognitive strategies while using ILM? Descriptive Statistics M= 4.54; SD= 0.52 Computer Self-Effficacy M= 4.34; SD= 0.64 M= 2.95; SD= 0.61 Cognitive Actions & Metacognitive Strategies M= 2.91; SD= 0.62 M= 2.89; SD= 0.54 M= 4.12; SD= 0.73 M= 2.72; SD= 0.59 Beginning Skills Advanced Skills File and Software Skills Planning Strategies Cognitive Actions Monitoring Strategies Regulating Strategies * Low-to-moderate: Mean value between 1.00 and 2.75 * Moderate-to-high: Mean value between 2.76 and 4.00 * Terminology used in Lawanto, Butler, Cartier, Santoso, Lawanto, and Clark, 2013

37 Addressing Research Question #1 How is students CSE related to cognitive actions and metacognitive strategies while using ILM? Relationships between CSE and Cognitive Actions and Metacognitive Strategies A significant positive correlation between CSE and cognitive actions, r (100) =.176, p <.05 A significant positive correlation between advanced skills component of the CSE and cognitive actions, r (100) =.185, p <.05 No significant correlation between CSE and overall metacognitive strategies, r (100) =.121, p =.115 Significant positive relationships between CSE and planning strategies, r (100) =.176, p <.05, and between beginning skills component of CSE and planning strategies, r (100) =.186, p <.05

38 Addressing Sub-Question #1 The Relative Importance of CSE with Regards to Its Contribution toward Students Cognitive Actions and Metacognitive Strategies To what degree CSE predicts Cognitive Actions: The three CSE components (i.e., beginning, advanced, and file and software skills) explained only 3.40% of the variance [R 2 =.034, F(3, 96) = 1.115, p =.347] (*) To what degree CSE predicts Planning Strategies: The three CSE components explained only 3.90% of the variance [R 2 =.039, F(3, 96) = 1.302, p =.278] (*) To what degree CSE predicts Monitoring Strategies: The three CSE components explained only 2.50% of the variance [R 2 =.025, F(3, 96) =.837, p =.477] (*) To what degree CSE predicts Regulating Strategies: The three CSE components explained negative 2.70% of the variance [R 2 = -.027, F(3, 96) =.147, p =.931] (*) (*) There are other factors that might contribute more to student cognitive actions, planning, monitoring, and regulating strategies while using ILM and should be investigated in future research.

39 Addressing Research Question #2 How do students plan (PLA) and monitor (MON) their cognitive actions, and regulate (REG) their monitoring strategies during learning with ILM? Cognitive Actions and Metacognitive Strategies of All Participants A series of paired t tests (2-tailed) was conducted to evaluate whether gaps between SRL features were significant. Gap Significant Difference? t and p values PLA > COG Yes t = 5.967, p <.001 COG < MON Yes t = , p <.001 MON = REG No t = 1.036, p =.303 The students did well in making plan before working on the modules. But the findings show they might struggled to execute their plan. It may be caused by the objectives of the modules presented to them. Interestingly they performed well in monitoring their actions and regulating/adjusting strategies based on the monitoring process.

40 Addressing Sub-Question #2: Quantitative Analysis Cognitive Actions & Metacognitive Strategies of High & Low CSE Groups Cognitive Actions and Metacognitive Strategies between High (n = 47) and Low CSE (n = 16) Groups. (Z = , p <.05) (Z = , p <.05) (Z = , p <.05) (Z = , p <.05) (Z = , p <.05)

41 Eight Selected Cases Rationale: To gather a reasonable number of quantitative and qualitative data from the same subjects in investigating their cognitive actions and metacognitive strategies. Selection Procedure: Order screen-captured videos based on their duration Prioritize the videos with long duration Use a stratified sampling to represent school and CSE level Logan High School (n) InTech Collegiate High School (n) High CSE Student 2 2 Low CSE Student 2 2

42 Data Triangulation : Overview From Quantitative Data [SRCBL Questionnaire]: Mean Score Planning Str. * Cognitive Act. Monitoring Str.* Regulating Str. High CSE Low CSE Note: * = significant difference From Qualitative Data [Screen-Captured Videos]: Duration of strategies Planning Str. Cognitive Act. Monitoring Str. Regulating Str. * High CSE 30 min., 38 sec. 137 min., 3 sec. 56 min., 31 sec. 47 min., 28 sec. Low CSE 35 min., 05 sec. 120 min., 13 sec. 47 min., 04 sec. 20 min, 19 sec. Note: * = significant difference From Qualitative Data [Interview]: Issues Gathered High CSE Planning Str. Cognitive Act. Monitoring Str. Regulating Str. Similar: Read materials & instructions first Organized More elaborative Similar: Check Low CSE Trial & error Less elaborative progress first, then try other strategies.

43 Addressing Sub-Question #2: Quantitative Analysis Cognitive Actions & Metacognitive Strategies of High & Low CSE Selected Cases Cognitive Actions & Metacognitive Strategies of High (n = 4) & Low (n = 4) Selected Cases. (Z =.018, p <.05) (Z =.020, p <.05) Both high and low CSE selected cases indicated no significant differences between PLA & COG, COG & MON, and MON & REG.

44 Addressing Sub-Question #2: Qualitative Analysis Examples of Cognitive Actions & Metacognitive Strategies Profiles Case #1: Andy High CSE REG: Trying other strategies MON: Checking answer; Trying to review the guidance COG: Reading the learning materials PLA: Reading the learning materials COG: Matching the objects with basic Boolean expressions Case #5: Earl Low CSE

45 Addressing Sub-Question #2: Qualitative Analysis Cognitive Actions and Metacognitive Strategies of Selected Cases Frequency of Strategy Changes While Using the Modules between High and Low CSE Groups The high CSE group changed their strategies more often that did the low CSE group on all modules. Frequency of Strategy Changes per Group Group Boolean Logic * Minimum Spanning Tree * Modeling Using Graphs * High CSE Student Low CSE Student Note: * = significant difference Frequency of Strategy Changes per Student Group Boolean Logic * Minimum Spanning Tree Modeling Using Graphs High CSE Student Low CSE Student Note: * = significant difference

46 Addressing Sub-Question #2: Qualitative Analysis Issues Gathered from Interviews No. Issue 1. Previous experience in using a computer helps students to use the interactive learning module. 2. Strategy of preparing to find solutions for the task. 3. Strategies to carry out plans while using the ILM. 4. Strategies used to detect any errors in solving the task or problem. 5. Strategies to fix any errors in solving a task or problem. 6. Success parameters of using the ILM according to the students. 7. Aspects of ILM that students like and dislike the most. Comparison Similar previous experience helped them Similar read materials & instructions Different high CSE organized; low CSE trial & error Different high CSE more elaborative; low CSE less elaborative Similar see the error first Different everyone has different perspectives Different high CSE feedback mechanism; low CSE interface

47 Implications (1/2) This research has implications for self-regulated learning researchers, teachers, and interactive learning module developers. No CSE component significantly predicts cognitive actions and metacognitive strategies. Self-regulated learning researchers may consider identifying other factors or motivational constructs, such as intrinsic and extrinsic motivations, that may be significant predictors toward students cognitive actions and metacognitive strategies while engaged in interactive learning modules.

48 Implications (2/2) It was indicated that the students found planning to be important, but they did not translate the planning into actions. The teachers may need to: 1. explain the introduction to the concepts before allowing the students to use the module; 2. encourage the students to read the objectives of activities on the modules carefully before executing their plans; and 3. have one or more teaching assistants to help him or her in responding any question raised by students while working with the module. Since the users computer self-efficacy is varied, the developers should make the modules more easier to navigate and graphics are created based on a real-world example.

49 Recommendations (1/2) First, this study only analyzed 100 datasets of the participating students. Larger participants from different schools may improve the generalizability of results of similar studies. Second, the way high-school students work may influence the results. The teacher presence during the data collection process could help the students to focus working on the modules. Third, the nature of this study is descriptive study. Experimental study can be conducted to see whether modified interactive learning modules can improve either computer selfefficacy or improve cognitive actions and metacognitive strategies.

50 Recommendations (2/2) Fourth, a method used in this study may be beneficial for other studies in computer-based learning environments. Fifth, while CSE questionnaire had high Cronbach s Alpha scores, SRCBL questionnaire had relatively low reliability scores for planning strategies. Analyzing more than one type of interactive learning module or computerbased learning environment may help in improving the items of the questionnaire.

51 Question and Answer