Exploring Revisiting in an Online Collaborative Learning Environment

Transcription

1 1. Objectives Exploring Revisiting in an Online Collaborative Learning Environment Lesley Wilton & Clare Brett Ontario Institute for Studies in Education, University of Toronto, Canada As online course offerings rise exponentially, and new technologies offer comprehensive tracking of online activities, examination of measures like rereading or revisiting become increasingly important. This study looks at the educational value students find in revisiting other s entries in online collaborative discussions. Data suggests that amounts of and purposes for revisiting vary with overall levels of reading and writing participation. We identified 3 clusters based on online activity and describe the characteristic activity patterns for each. We also found a positive relationship between revisiting and learning outcomes and describe the characteristics of highly revisited entries. Implications for instruction and for further research are discussed. In light of the increased offerings of online learning courses, recent research in online learning has started to investigate more deeply the role of reading and revisiting entries in online collaborative discourse to develop a more comprehensive understanding of how online reading activities may contribute to learning (Wise, Marbouti, Speer & Hsiao, 2011; Wise, Marbouti, Hsiao & Hausknecht, 2012a; Brett & Wilton, 2013; Hewitt, Brett & Peters, 2007; Qiu, Hewitt & Brett, 2012). In this paper we build on this earlier work by looking specifically at the role of online revisiting patterns of graduate students in fully online courses. Revisiting may be a potential indicator of important elements of learning. For example, entries that are revisited a lot may signify important or useful content and rereading may be a useful indicator of where student attention is focused within the course conference. In addition, revisiting may also indicate points where students are either struggling with, or focusing on, particular ideas. Understanding rereading may therefore provide important information for instructors about levels of student understanding (Wise, Perera, Hsiao, Speer & Marbouti, 2012b). 2. Perspectives and theoretical framework Participatory activity is viewed by sociocultural theorists, including Vygotsky (1986), Wenger (1998) and Lave & Wenger (1991), as the central mechanism for learning. Learner participation in online learning environments has long been considered an important element supporting learning in such environments (e.g. Hrastinski, 2008; Rovai, 2002; Swan, Shea, Fredericksen, Pickett, Pelz & Maher, 2000). However, participation has largely been defined in terms of written participation in discussions--sometimes in terms of quantity, such as number of messages, and in other instances looking at frequencies of units such as words and phrases or ideas (Hakkarainen & Palonen, 2003). Other studies have looked at quality of posts within classification schemes such as asking questions, providing information, etc. (e.g. Carr, Cox, Eden & Hanslo, 2004). Measures of participation such as learner perceptions, message length, or logins are also frequently used. However one of the less used measures is that of reading messages. In fact, a review by Hrastinski (2008) found only three studies that used reading (defined as opening posts)

2 EXPLORING REVISITING Page 2 as a measure. Online reading, when it has been researched, has been conceptualized variously as an indicator of online engagement (Brett, 2004) and as an element of online listening (Wise et al., 2011; Wise et al., 2012a). Online reading has been measured through metrics such as scan rate (Hewitt, Brett & Peters, 2007), or note opening, and online listening (which posts students attend to, when and how) (Wise et al., 2011; Wise et al., 2012a). Literature taking a broadly social constructivist stance has long emphasized the importance of discourse and collaboration to learning (e.g. Garrison, Anderson & Archer 2000; Harasim, 2012; Swan & Shih, 2005). Generally described, learners build upon their own reflections and the opinions of others by engaging in online dialogue. This process is believed to contribute to the individual s processing of material. Such research has influenced the design of online learning courses that typically value online discussions, particularly written postings. Often instructors provide some portion of grades based on such evidence. It seems clear that discussion-based learning in an online environment involves reading others postings. However, student reading and rereading of others responses typically appears to be assumed rather than tracked specifically and thus there has been a lack of investigation of the potential importance to learning of reading other students postings. Reading may be less studied because it has been harder to accurately measure and is less visible to the instructor. It is harder to assign grades because clear evidence of reading is difficult to capture compared to evaluating student postings. Many learning management systems can capture views (whether a message has been opened) but cannot account for those who may open posts without actually reading them. In research, some solutions to these problems have been found through metrics such as Scan rate (Hewitt, Brett & Peters, 2007) and through treating online reading more contextually, conceiving of it as being part of listening behaviours (Wise, Speer, Marbouti & Hsaio, 2013b) defined as taking in the externalization of others (p. 5). Online listening while importantly different from listening in a face-to-face conversation may be valuable because it can be considered an active form of participation in online discussions (Wise et al., 2013b). Even less studied than online reading or listening is the notion of rereading or revisiting. Wise, Hausknecht & Zhao (2013a) did find a positive relationship between rereading other students posts and responsiveness and suggest that the richer end of this spectrum tends to occur when posts are attended to multiple times (p. 6). The authors also suggested that the [n]umber of reviews of other students posts was a positive predictor of more responsive posts (p. 6). The research questions guiding this paper therefore contribute to expanding this literature, and they are: 1. What seem to be the characteristics of revisiting in relation to reading and writing of entries? 2. Are there particular qualities possessed by entries that are widely revisited? 3. Is there any relationship between particular patterns of revisiting behaviors and outcomes?

3 EXPLORING REVISITING Page 3 3. Methods This study uses a mixed methods design to analyze the online rereading activity of a total of 64 participants across four fully online graduate courses conducted over a two year period. These courses were facilitated in PeppeR, a computer conferencing program developed at the Ontario Institute for Studies in Education at the University of Toronto. This program has a variety of features such as public discussion threads, private or group messaging, and the facilitation of reading functions such as selecting a split screen, contents or titles display. Tracking detail is collected by the system on entries, events and user activities. In all four courses, students were required to provide an entry in the Class Biographies Conference. A Learning Journal was a requirement in each course, worth 25% of the course grade. In two of the courses, a Theory of Learning entry, posted at the beginning of the term and reviewed at the end of the term, addressed questions intended to document any changes in understanding. These entries comprised 15% of the course grade. All assignments were posted publicly in the class conference. 4. Data sources and analysis The quantitative measures used include automatically collected tracking data on all online activities and the analyses of these measures using cluster analysis and bivariate correlations (Field, 2005; Green & Salkind, 2011). Activities related to 6026 entries in four databases were categorized and 23 relevant variables were selected for analysis. Variables were categorized according to Temporal, Collaborative/Social, Writing Quality, Revisitation and Online Participation Characteristic dimensions. Only a small subset of variables is reported on here. For all 64 cases, participants were clustered into three categories based on reading and writing patterns, with an SPSS reported good quality silhouette measure of cohesion and separation (0.6). Cluster Number of Notes Written Percentage of Total Notes Read Number of Cases 1. Prolific Writing/Broad Reading (PWBR) 86 to % to 100% Efficient Writing/Broad Reading (EWBR) 24 to 78 64% to 99% Efficient Writing/Selective Reading (EWSR) 18 to 89 17% to 62% 24 Table 1 Cluster descriptions Individual notes that were highly revisited were identified and content qualities were analyzed according to an established coding scheme (Wise et al., 2013a). Wise et al. (2013a) developed a coding scheme (see Table 2) for speaking variables based on a combination and adaptation of prior schemes and models by Hara et al. (2000), Knowlton, 2005, Pena-Shaff & Nichols (2004), Weinberger & Fischer (2006) and Wise et al. (2012d) ( p. 3). This established coding scheme was adopted for this preliminary examination. The qualitative measures include responses to a 15-question pre-tested survey collecting open-ended and Likert-scale data on rereading activity and individual

4 EXPLORING REVISITING Page 4 case studies from cluster groups identified through the statistical analyses. Table 2 Wise et al. s (2013a, p. 3) overview of coding scheme for speaking variables 5. Results and discussion To answer the first research question: What seem to be the characteristics of revisiting in relation to reading and writing of entries, we analyzed the levels of writing and reading behaviors across the 64 cases and found three clear clusters as outlined in Table 1. The first cluster, which we have called, for now, Prolific Writing/Broad Reading (PWBR), consists of students who wrote between 86 to 241 notes, and read between 42% and 100% of the notes in any course. The second cluster, called Efficient Writing/Broad Reading (EWBR), consists of students who wrote between 24 and 78 notes, and read between 65% and 99% of the notes in any of the 4 courses. The third cluster, called Efficient Writing/Selective Reading (EWSR), consists of students who wrote between 18 and 89 entries, and read between 17% and 62% of the notes in any particular course. Cluster Hours Online M Who is Revisiting? Student Notes M Teacher Notes M Who is being Revisited? Rereads by Other Students M Final Marks Range 1 (PWBR) (EWBR) (EWSR) Table 3 Reading and writing patterns in relation to revisiting characteristics 1 Example of significant result* Links to students notes from other notes Student Notes reread r =.44, p <.05 Rereads by Other Students r =.63, p <.01 Revisiting Teachers Notes r =.46, p <.05 1 *Note: These are examples of interesting significant differences we found between the groups. Space constraints limit our discussion of all the variables used in this analysis.

5 EXPLORING REVISITING Page 5 It should be noted that we are finding the naming of these patterns to be problematic. We do not like to refer to these behaviors by making assumptions about the students themselves. Therefore, we have avoided using terms like lurkers, free-loaders or nonpublic participants (Dennen, 2008; Nonnecke & Preece, 2003). Instead we have tried to focus on the behavioral patterns, such as the behavior of Selective Reading/Efficient Writing. The acronyms of PWBR (those who write more and read more), EWBR (those who write less and read more) and EWSR (those who write less and read less) are not easy to recall and do not rhyme off the tongue (poo-burr, eww-burr and user). We also considered referring to those who read or write less, as Reading-lite or Writing-lite. For example, an EWSR behavior would become a Writing-lite/Reading-lite behavior. We are still seeking the right terms to use to describe these clusters of behaviors. In any case, we have summarized these data by cluster rather than individually due to space restrictions and highlighted some of the significant relationships with asterisks. Responses to the survey questions revealed that all students said that revisiting entries was important or very important to their learning. Examples of responses from students to the question about why they reread are revealing: From PWBR (write/read more) Cluster 1: I would revisit or reread student-generated notes to clarify course concepts, to build on their notes (i.e. so I would refer back to their note to make sure I was interpreting them correctly), or I would refer to their notes for assignment purposes. From EWBR (write less/read more)cluster 2: One of the other reasons I revisited a previously read entry was because as the course evolved, my knowledge evolved (and in some cases my beliefs began to change)...and when that happened, it twigged memory of a post that had been put up earlier in the course, that I wanted to rethink and assimilate with my new understanding. From EWSR (write less/read less) Cluster 3: I often revisited notes to while formulating my own responses, to check that I understood what others had communicated, to search for a recommended article. One student in particular was always in the good habit of using quotes. I often re-read that student's messages to use the quotes. The data and illustrative comments reveal the importance of rereading for a variety of learning-related reasons. The second research question: Are there particular qualities possessed by entries that are widely revisited?, involved an analysis of the top 62 student-generated revisited notes from one course (reread at least once by more than half the class,), which yielded 36 revisited entries in categories of biography (5), learning journal (11), external resources provided by a student (2), learning theory (3), response to a moderator question (13) or response to a post (2) within the weekly conference discussion. The top 15 revisited response posts were coded for content qualities based on the speaking variables detailed in Table 2 by Wise et al. (2013a), of Discursiveness, Content and Reflectivity. The majority of these entries contained complex argumentation, synthesized others notes and/or developed ideas. The rest contained simple argumentation and referred to others ideas or course materials. All of the notes demonstrated deep group reflection and one-third of these entries demonstrated a level of elicitation.

6 EXPLORING REVISITING Page 6 To answer the third research question: Is there any relationship between particular patterns of revisiting behaviors and outcomes, we looked at final grades and amount of revisiting by cluster. An analysis of all students (N=64) shows a significant correlation between Final Grades and Student Notes Reread (r =.56, p <.001), Teachers Notes Reread (r =.65, p <.001) and Rereads by Other Students (r =.49, p. <.001). Further breakdown by cluster reveals specific patterns. Who is Revisiting? Who is being revisited? Student Notes Teacher Notes Rereads by Other Students Final Grade PWBR (1): r =.53, p <.05 EWBR (2): n.s. EWSR (3): r =.66, p <.001 PWBR (1): r =.55, p <.05 EWBR (2): n.s. EWSR (3): r =.68, p <.005 PWBR (1): n.s. EWBR (2): n.s. EWSR (3): r =.61, p <.005 Table 4 Relationship between revisiting and final grade by cluster 6. Scientific or scholarly significance of the study or work The revisiting patterns of the discussion entries and responses to survey questions support the prevalence of, and importance of rereading for learning. Some interesting findings are the differential relationship of revisiting rates to final grades for the EWSR (Efficient Writing/Selective Reading) cluster which does suggest that rereading may be an important strategy for those who participate in less intensive ways. Another notable finding was the amount of revisiting in areas of the course like biographies and individual learning journals that were not directly assessed for marks. Participants survey comments suggest they use rereading of others learning journals to support their own reflections, and rereading others biographies to find partners for collaborative assignments and to maintain social connections with their program peers. As we continue this research we will be investigating some of the pedagogical implications of rereading, such as the potential for identifying realtime thresholds of highly read notes and giving that feedback to instructors about where students attention is focused. We will investigate how feedback on rereading patterns, such as those identified through our cluster analysis, may also provide useful information for instructors, and possibly students. This micro-analytic study contrasts with the current emphasis on large-scale online learning initiatives like MOOCs. However, as new tools make it increasingly possible to innovatively track such detailed learner activities online we will be able to understand better how particular activities contribute to how people learn. Without such information gleaned at the microlevel layer of research, we will not be able to effectively and confidently design new online learning environments that emerge so rapidly in this area. References Brett, C. (2004). Offline factors contributing to online engagement. Technology, Pedagogy and Education, 13(1),

7 EXPLORING REVISITING Page 7 Brett, C. & Wilton, L. (2013, April). Online reading patterns and their relationship to online writing activity. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA, USA. Carr, T., Cox, G., Eden, A, & Hanslo, M. (2004). From peripheral to full participation in a blended trade bargaining simulation, British Journal of Educational Technology, 35(2), doi: /j x Dennen, V. P. (2008). Pedagogical lurking: Student engagement in non-posting discussion behavior. Computers in Human Behavior, 24, ) Field, A. (2005). Discovering Statistics Using SPSS: (and sex, drugs and rocknroll). Thousand Oaks, CA: SAGE Publications. Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), Green, S. B. & Salkind, N. J. (2011). Using SPSS for Windows and Macintosh: Analyzing and Understanding Data. Upper Saddle River, NJ: Prentice Hall. Hakkarainen, K. & Palonen, T. (2003). Patterns of female and male students participation in peer interaction in computer-supported learning. Computers and Education, 40( 4), Harasim, L. (2012). Learning Theory and Online Technologies. New York, NY: Routledge. Hewitt, J. (2003). How habitual online practices affect the development of asynchronous discussion threads. Journal of Educational Computing Research, 28(1), Hewitt, J. (2005). Toward an understanding of how threads die in asynchronous computer conferences. Journal of the Learning Sciences, 14(4), Hewitt, J., Brett, C. & Peters, V. (2007). Scan Rate: A new metric for the analysis of reading behaviors in asynchronous computer conferencing environments. American Journal of Distance Education, 21(4), Hewitt, J., & Scardamalia, M. (1998). Design principles for distributed knowledge building processes. Educational Psychology Review, 10(1), Hrastinski, S. (2008). What is online participation? A literature review. Computers & Education, 51, Lave, J., & Wenger, E. (1991). Situated learning. Legitimate peripheral participation. Cambridge: Cambridge University Press. Nonnecke, B. & Preece, J. (2003). Chapter 6: Silent Participants: Getting to know lurkers better. In C. Lueg & D. Fisher (Eds.) From Usenet to CoWebs: Computer Supported Cooperative Work, pp Qiu, M., Hewitt, J., & Brett, C. (2012). Online class size, note reading, note writing and collaborative discourse. International Journal of Computer-Supported Collaborative Learning, 7(3), doi: /s Rovai, A. P. (2002). Sense of community, perceived cognitive learning, and persistence in asynchronous learning networks, The Internet and Higher Education, 5(4), doi: /S (02) Swan, K. & Shih, J. F. (2005). On the nature and development of social presence in online course discussions. Journal of Asynchronous Learning Networks, 9(3),

8 EXPLORING REVISITING Page 8 Swan, K., P. Shea, E. Fredericksen, A. Pickett, W. Pelz, and G. Maher. (2000). Building knowledge building communities: consistency, contact and communication in the virtual classroom. Journal of Educational Computing Research, 23(4): Wenger, E. (1998). Communities of practice. Learning, Meaning and Identity. Cambridge: Cambridge University Press. Wise, A.F., Hausknecht, S., Zhao, Y. (2013a). Relationships between listening and speaking in online discussions: An empirical investigation. In N. Rummel, M. Kapur, M. Nathan & S. Puntambekar (Eds.) Proceedings of the 10th International Conference on Computer Supported Collaborative Learning Vol I (pp ). Madison, Wisconsin. Wise, A. F., Marbouti, F., Hsiao, Y. & Hausknecht, S. (2012a). A survey of factors contributing to learners listening behaviors in asynchronous online discussions. Journal of Educational Computing Research, 47(4), Wise, A. F., Marbouti, F., Speer, J., & Hsiao, Y. T. (2011). Τowards an understanding of listening in online discussions: A cluster analysis of learners interaction patterns. In H. Spada, G. Stahl, N. Miyake & N. Law (Eds.), Connecting computer supported collaborative learning to policy and practice: CSCL2011 conference proceeding (Vol. I), Long papers (pp ), International Society of the Learning Sciences. Wise, A. F., Perera, N., Hsiao, Y., Speer, J., & Marbouti, F. (2012b). Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course. The Internet and Higher Education, 15(2), Wise. A. F., Speer, J., Marbouti, F. & Hsiao, Y. T. (2013b). Broadening the notion of participation in online discussions: Examining patterns in learners online listening behaviors. Instructional Science, 41(2), Vygotsky, L. (1986). Thought and language. Cambridge, MA: The MIT Press.