Developng an IS for Montorng Student s Actvty n Onlne Collaboratve Learnng 1 Developng an Informaton System for Montorng Student s Actvty n Onlne Collaboratve Learnng Angel A. Juan, Thanass Daradoums, Javer Fauln, and Fatos Xhafa Abstract In ths paper we address the ssue of montorng students and groups actvty n onlne collaboratve learnng envronments. Ths ssue s especally mportant n the collaboratve e-learnng context, snce an effcent montorng process can provde valuable nformaton to onlne nstructors who may gude and support the development of collaboratve learnng projects. We have developed and tested an nformaton system model whch facltates the automatc generaton of weekly montorng reports derved from data contaned n server log fles. These reports provde onlne nstructors wth vsual nformaton regardng students and groups actvty, thus allowng for a quck and easy classfcaton of students and groups accordng to ther actvty level. Therefore, enttes wth a low actvty level are dentfed as soon as possble and just-ntme assstance can be establshed for them. Furthermore, nstructors can use these montorng reports to forecast potental problems such as students dropouts or possble conflcts nsde the groups due to unbalanced dstrbuton of tasks and take operatonal and tactcal decsons orented to avod them. Index Terms collaboratve learnng; onlne educaton; montorng students actvty; just-n-tme assstance I I. INTRODUCTION NFORMATION technologes offer new ways to communcate, collaborate and partcpate n learnng processes. Snce technology s changng the methods through whch educaton s delvered, colleges and unverstes across the world are confrontng several transformatons whch affect the nature of the courses and degree programs they offer. These technologcal nnovatons have also drven the growth of dstance learnng opportuntes, as students who are tme bound due to job or personal responsbltes or place bound due to geographc locaton or physcal dsabltes can now Manuscrpt receved October 25, 2007. Ths work has been partally supported by the Spansh Mnstry of Educaton under grants TSI2005-08225- C07-05 and EA2007-0310. A. A. Juan s wth the Open Unversty of Catalona, Barcelona 08018 SPAIN (correspondng author, phone: +34 933 263 627; fax: +34 933 568 822; e-mal: ajuanp@uoc.edu). T. Daradoums s wth the Open Unversty of Catalona, Barcelona 08018 SPAIN (e-mal: adaradoums@uoc.edu). J. Fauln s wth the Publc Unversty of Navarre, Pamplona 31006 SPAIN (e-mal: javer.fauln@unavarra.es). F. Xhafa s wth the Techncal Unversty of Catalona, Barcelona 08034 SPAIN (e-mal: fatos@ls.upc.edu). access courses and degree programs at ther convenence. Because of the rapd growth of dstance and global educaton, e-learnng models are currently practced wdely all over the world. As some authors pont out, e-learnng models can provde hgh qualty educatonal offerngs at the same tme they allow for convenent and flexble learnng envronments wthout space, dstance or tme restrctons [1]. Moreover, educatonal technologes facltate the shftng from a tradtonal educatonal paradgm centered on the fgure of a masterful nstructor to an emergent educatonal paradgm whch consders students as actve and central actors n ther learnng process. In ths new paradgm students learn, wth the help of nstructors, technology and other students, what they wll potentally need n order to develop ther future academc or professonal actvtes. The nstructor s role s, therefore, movng from one related to a knowledge transmsson agent to another related to a specalst agent who desgns the course, gudes, asssts and supervses the student's learnng process [2], [3]. In onlne learnng envronments lke Moodle, WebCT or BSCW, nstructors provde students wth course core materals and, addtonally, wth complementary learnng resources such as web lnks, overhead presentatons, software-based smulatons, self-assessment tests, research artcles, Java applets, etc. At the same tme, they set up ndvdual or collaboratve learnng actvtes to gude the learnng process, provdng assstance at dfferent levels whle moderatng and supportng dscussons n ether small group or class forums. Onlne students, n turn, are encouraged to use these resources, partcpate n learnng actvtes and engage n collaboratve tasks where they have the opportunty to express deas, dscuss course topcs and work out complex delverables. II. MONITORING ACTIVITY IN COLLABORATIVE E-LEARNING Despte the benefts that Internet-based educaton can offer both to students and nstructors, t also presents some mportant challenges. Typcally, any type of dstance educaton program presents hgher dropout rates than more conventonal programs [4]. The nature of dstance educaton can create a sense of solaton n learners, and students can feel dsconnected from the nstructor, the rest of the class, and even the nsttuton. It s necessary, then, that nstructors provde just-n-tme gudance and assstance to students
Developng an IS for Montorng Student s Actvty n Onlne Collaboratve Learnng 2 actvtes and also that they provde regular feed-back on these actvtes. Furthermore, communcaton among students should also be facltated and promoted by nstructors who should encourage students partcpaton n the web spaces devoted to that functon. Unfortunately, t s very dffcult and tme consumng for nstructors to thoroughly track all the actvtes performed by each ndvdual student n these e-learnng envronments. It s even much more complex to fgure out the nteractons takng place among students and/or groups of students, to dentfy actors groups leaders and followers, to detect students that are lkely to dropout the course, or to perceve possble group nternal conflcts or malfunctons before t gets too late to effcently manage these problems. Montorng students and groups actvty can help to understand these nteractons and forecast these potental problems whch, n turn, can gve mportant clues on how to organze learnng exercses more effcently and thus acheve better learnng outcomes [5], [6]. Montorng reports can be used by nstructors to easly track down the learners onlne behavor and group s actvty at specfc mlestones, gather feedback from the learners and scaffold groups wth low degree of actvty. Montorng has a tme dmenson, that s, nstructors have to know both the groups and students actvty performance as the learnng process gets developed. The montorng process can thus be a means for nstructors to provde just-n-tme assstance accordng to groups and students necesstes. IV. OBJECTIVES AND SCOPE OF OUR WORK As n any other unversty offerng onlne programs, n the case of the Open Unversty of Catalona (UOC, http://www.uoc.edu), nstructors need non-ntrusve and automatc ways to get feedback from learners progress n order to better follow ther learnng process and apprase the onlne course effectveness. Desgnng effcent montorng tools for onlne collaboratve envronments s certanly a complex task. Ths s partly due to a lack of practcal models that had been already tested n real stuatons nvolvng consderable number of students, groups and nstructors. Therefore, the man goal of ths work s to develop, mplement and test a practcal nformaton system that allows nstructors at the UOC to effcently montorng students and groups actvty n collaboratve e-learnng courses. Even when the model presented n ths paper has been desgned to meet the UOC specfc requrements, t can serve as a conceptual framework that can be used for trackng groups and ndvduals actvty n any e-learnng envronment. In partcular, t can be especally useful n those collaboratve e-learnng courses that: (a) span over one or more semesters, (b) nvolve a large number of groups and students that need to develop a contnuous and ntense collaboratve actvty, and (c) pursue specfc academc goals regardng students actve partcpaton, low dropout rates and avodance of groups malfuncton. III. EXISTENT RESEARCH ON THE ISSUE Due to ts mportance, several works n the Computer Supported Collaboratve Learnng lterature, and more especally those related to onlne collaboratve learnng, have addressed the montorng ssue from dfferent perspectves, yet they all provde a very lmted scope and do not rase most practcal ssues. Rather, they are concerned wth conceptual aspects of montorng [7] [9]. There s also a wde varety of proposed methods to montorng group and ndvdual actvty n onlne collaboratve learnng. These methods nclude statstcal analyss, socal network analyss, and montorng through shared nformaton and objects [10] [12]. Moreover, there exst some dfferences as regards the sources of nformaton used for montorng: log fles of synchronous and asynchronous communcaton, bulletn boards, electronc dscusson nformaton reports, etc. In general, though, the montorng and evaluaton of learners actvty n onlne learnng envronments s stll an mportant topc n the feld of open and dstance educaton. As some authors recognze, nstructors partcpatng n onlne learnng envronments have very lttle support by ntegrated means and tools to montor and evaluate students actvty [13], [14]. As a consequence, ths montorng process consttutes a dffcult task whch demands a lot of resources and expertse from educators. V. THE COLLABORATIVE E-LEARNING SCENARIO AT THE UOC In order to desgn our montorng system at the UOC, we have consdered a common scenaro where groups of students have to develop long-term projects, whch are problemsolvng collaboratve practces. Such projects are organzed n terms of several phases, each of them correspondng to a target goal. The nstructonal desgn of each target goal ncludes several learnng tasks, adequately lnked to each other, whch students should carry out ndvdually such as readngs or collaboratvely such as group actvtes and exercses n order to acheve the correspondng goal. In addton, the desgn of some target goals also nvolves the realzaton of specfc asynchronous debates at group or class level, amng at decson takng on a set of specfc questons. These projects are carred out n the scope of several dstance learnng undergraduate courses whch typcally run over a perod of 15 weeks. Each of these courses nvolves one academc coordnator, several nstructors one for each vrtual class and the class of students about 50 per class dstrbuted among dfferent onlne groups wth 3 to 5 members each (Fg. 1).
Developng an IS for Montorng Student s Actvty n Onlne Collaboratve Learnng 3 Fg. 1. Collaboratve e-learnng Scenaro at the UOC The web platform that we use to develop collaboratve e- learnng courses at the UOC s the Basc Support for Cooperatve Work (BSCW) system (http://bscw.ft.fraunhofer.de/), a groupware tool that enables asynchronous and synchronous collaboraton over the web [15]. Ths system, lke any other smlar onlne collaboratve envronment, offers shared workspaces that groups can use to store, manage, jontly edt and share documents, realze threaded dscussons, etc. Addtonally, the BSCW server keeps log fles whch contan all the actons (events) performed by group members on shared workspaces, as well as detaled nformaton about these actons: user dentfcaton, event type, tmestamp, assocated workspace, affected objects, etc. Even though most e-learnng envronments offer some smple montorng tools, they are very lmted for practcal purposes and do not meet nformaton necesstes of onlne nstructors [16]. As a matter of fact, developers of the BSCW system recognze the need for powerful montorng models and tools. To ths end, our model wll make use of the BSCW log fles to generate vsual reports that summarze relevant nformaton on students and groups actvty. VI. OUR COLLABORATIVE E-LEARNING SCENARIO Fg. 2 shows the global scheme of the montorng system that we have developed and tested at the UOC. The general functonng of ths model s explaned below: 1) Students perform actvtes n the web collaboratve spaces assocated to ther workng group: they post or read notes n forums, sent or read e-mals, upload or download documents, manage folders and fles, etc. Each of these actvtes can be consdered as an event of a certan type whch has been developed by a partcular student at a certan tme and web space. 2) Events generated by students are regstered n log fles at the web server whch supports the e-learnng envronment. In our case ths server runs the BSCW web platform, but other platforms such as Moodle or WebCT would mantan smlar log fles. 3) A specfc-purpose Java applcaton, called EICA, s used to automatcally read and process new ncomng log fles and to store the extracted data nto a unque persstent database n the correspondng server. Note that EICA could be adapted to read and process log fles from web platforms other than BSCW, such as the ones cted before. 4) Database fles are then processed by SAMOS, whch s an Excel/VBA applcaton developed at the UOC. SAMOS uses Excel numercal, graphcal and programmng capabltes to generate weekly reports whch summarze group and student actvty levels n a graphcal manner [17], [18]. The detals regardng the desgn of these reports, whch represent the core part of our model, are explaned n the next secton. 5) The SMTP server automatcally sends out these reports to nstructors by e-mal. 6) Instructors receve these reports and analyze them, lookng for groups and students whch seem to be at rsk,.e.: students wth low actvty levels whch makes them lkely to be non-partcpatng students and possble dropout students, and groups wth low actvty levels whch makes them lkely to be malfunctonng groups. 7) These results are then combned and contrasted wth the qualtatve self-, peer- and group evaluaton reports whch are generated by the students themselves. 8) Once the groups and students at rsk have been detected, nstructors contact them to offer specfc gudance and support towards the best development and completon of ther projects. The specfc actons to be performed by nstructors depend on the characterstcs of the current learnng actvty and the type of problem detected. In any case, the mportant pont here s that nstructors become aware of the low actvty problems as soon as they appear and, therefore, they can react on tme, whch adds value to ther role as supervsors of the learnng process. 9) Ths way, students and groups at rsk, receve just-ntme gudance and support to enhance and contnue ther ndvdual or collaboratve work. VII. THE SAMOS MONITORING REPORTS Regardng the weekly montorng reports, our goal was to desgn a small set of graphs that were easly and quckly understood by nstructors, so that they dd not have to nvest extra tme n analyzng data. These graphs should contan only crtcal nformaton about groups and students actvty levels. Furthermore, they should provde nstructors wth a rough classfcaton for each knd of enttes groups and students accordng to ther correspondng actvty levels. Specfcally, they should allow nstructors to easly dentfy those groups and students that were bound to mantan extremely low actvty levels, snce those enttes are lkely to need just-n-tme gudance and assstance.
Developng an IS for Montorng Student s Actvty n Onlne Collaboratve Learnng 4 Fg. 2. General Scheme of our Montorng Model Smlarly, these graphs should also provde nformaton about the hstorcal evoluton of each group s actvty wth respect the rest of the class groups, as well as nformaton about the hstorcal evoluton of each student s actvty wth respect to the rest of group members. Havng these consderatons n mnd, we desgned the followng four charts: (a) a groups classfcaton graph, (b) a students classfcaton graph, (c) a group s actvty-evoluton graph, and (d) a student s actvty-evoluton graph. Each of these charts s descrbed next: Groups Classfcaton Graph: Ths chart (Fg. 3) s a scatterplot of the followng two varables: X = average number of events per member that have been generated by group durng ths (current) week ( = 1, 2,..., n), and Y = average number of events per member that have been generated by group durng a course average week. The plot also ncludes the straght lnes x = x and y = y, whch dvde the graph n four quadrants, Q1 to Q4. That way, ponts n Q1 can be seen as headng groups snce ther actvty levels are above the two actvty means current week and course average week ; ponts n Q2 can be consdered as lowerng groups, snce even when hstorcally ther actvty level has been above the actvty level for an average week, ther current actvty level s below the average; ponts n Q3 represent those groups whch are below the two actvty means current and hstorcal and, therefore, they can be consdered as groups at rsk, snce they are the most lkely to suffer from low task contrbuton, group malfunctonng, lack of socal coheson and eventually from students dropouts; fnally, groups n Q4 can be seen as mprovng groups, snce even though ther actvty level has been hstorcally below the mean, ther level has been above the mean durng the current week, so they are expermentng some mprovement n ther actvty level maybe as a consequence of just-ntme gudance by the nstructor. Note that, as the dstance between a pont and any of the straght lnes ncreases, more sgnfcant wll be the former nterpretatons. Fg. 3: Groups Classfcaton Graph Students Classfcaton Graph: Ths chart s smlar to the one before. The only dfference s that now the ponts wll represent students nstead of groups. Therefore, ths graph allows for an easy dentfcaton of those students at rsk that s, students whose actvty levels are below the current week average and below the hstorcal week average. Analogously to what happened wth groups, students can also be classfed as mprovng students, lowerng students or headng students dependng on the quadrant they belong to. Group s Actvty-Evoluton Graph: There s one of these charts for each group of students (Fg. 4). Ths way, for any gven group the correspondng chart shows:
Developng an IS for Montorng Student s Actvty n Onlne Collaboratve Learnng 5 (a) a tme seres representng the group s hstorcal evoluton that s, the number of events per member generated by the group durng each week, (b) two smoothed bands whch provde the lower (LQ) and hgher (HQ) quartles assocated to the dstrbuton of the events generated by each group durng the current week ths way, t s mmedate to check whether the group s performng above the thrd quartle, below the frst one, or n between, and (c) an exponentally smoothed lne, usng a smoothng factor of ω = 0.3 [19], that gves a forecast for the next week group s actvty. Ths chart allows the nstructor not only to follow but also to predct the group s evoluton throughout the course. Fgure 4: Group Actvty Graph Group Members Accumulated Actvty Graph: There s also one of these charts for each group. Gven a group, the correspondng graph shows the percentage contrbuton of each member wth respect to the total actvty developed by the group untl the current week (Fg. 5). VIII. MODEL VALIDATION In order to test whether or not the nformaton provded by our nformaton system may nfluence groups and students performance n collaboratve courses at the UOC, we developed the followng experment: at the begnnng of the second semester of the 2006/07 academc course, a random sample of sze 40 was drawn from the populaton of groups that were partcpatng n any collaboratve e-learnng course. Durng the semester, nstructors of these selected groups were provded wth weekly reports generated by our system, so that they could detect students and groups at rsk and provde them wth just-n-tme gudance and support. At the end of the semester, we calculated the followng ndexes: 1. Percentage of sampled groups whch fnshed ther project accordng to ts ntal specfcatons (PGF). 2. Percentage of sampled groups whch receved a postve evaluaton at the end of the semester (PGP). 3. Percentage of sampled groups whch expermented dropouts (PGD) that s, some of the group members abandoned the course durng the semester. Moreover, we used hstorcal data from past semesters to obtan the before-samos populaton percentages for these ndexes, p 0 ( = 1, 2,3 ). Then, for each selected ndex, we consdered the correspondng hypothess tests about the populaton proportons [20],.e.: H0 : psamos = p0 versus HA: psamos p0. Both percentages and results for these tests are shown n Table I. TABLE I HYPOTHESIS TESTS ABOUT THE POPULATION PROPORTIONS p p 0 SAMOS Index (data) (n = 40) 95% CI p-value 1. PGF 55% 75% (30) (0.59, 0.87) 0.011 2. PGP 49% 65% (26) (0.48, 0.79) 0.056 3. PGD 43% 25% (10) (0.13, 0.41) 0.025 Fgure 5: Group Members Actvty Graph From ths chart, group leaders and group nonpartcpatng members can be easly dentfed, allowng nstructors to mmedately actvate polces amng at preventng negatve stuatons such as neffcent or unbalanced dstrbuton of group tasks or student abandonment. Usng a standard sgnfcance level, α = 0.05, we could conclude from the correspondng p-values that those tests assocated wth ndexes 1 and 3 were sgnfcant. In other words, statstcal evdence supports the dea that the nformaton provded by our system contrbuted to sgnfcantly enhance the PGF and PGD ndexes n collaboratve e-learnng courses offered at the UOC. IX. FUTURE WORK In order to mplement and test our system nformaton model, we developed two ntal versons of the computer programs EICA and SAMOS. These versons are partally
Developng an IS for Montorng Student s Actvty n Onlne Collaboratve Learnng 6 based on propretary software and, furthermore, they present some lmtatons regardng the generalzaton of our approach to web platforms other than BSCW. Our current work deals wth the development of two open source versons of both programs. These new versons wll be completely based on Java and PHP and, moreover, they wll be able to read and process log fles from several e-learnng platforms. X. CONCLUSION Two major related problems n dstance learnng courses are: (a) to assure that students wll reach a satsfactory level of nvolvement n the learnng process, and (b) to avod hgh dropout rates caused by the lack of adequate support and gudance. These problems are even more crtcal n collaboratve e-learnng scenaros, where ndvdual dropouts or ndvdual low level nvolvements could force groups to loose coheson, face anxety or spend too much tme and efforts to rearrange ther actvtes, whch may cause a slowdown or even a breakdown of the group s actvty. Montorng students and groups actvty can be very useful to dentfy non-partcpatng students or groups wth unbalanced dstrbuton of tasks. Ths dentfcaton process, n turn, allows nstructors to ntervene whenever necessary to ensure and enhance student s nvolvement n the collaboratve learnng process. The montorng system model presented n ths paper has been successfully used to track groups and students actvty n several undergraduate onlne courses offered at the Open Unversty of Catalona. These courses nvolve long-term, project-based collaboratve learnng practces. Weekly montorng reports are used by nstructors to easly track down the students and groups actvty at specfc mlestones, gather feedback from the learners and scaffold groups wth low degree of actvty. Our nformaton system model has proved to be an nnovatve montorng tool for our onlne nstructors, snce t provdes them wth prompt and valuable nformaton whch adds value to ther role as supervsors of the learnng process and allows them to offer just-n-tme gudance and assstance to students and groups. In our opnon, ths model can serve as a practcal framework for other unverstes offerng collaboratve e-learnng courses. Internatonal Workshop on Groupware CRIWG 05. Pernambuco, Brazl, pp. 284-291, 2005. [8] G. Joyes, and P. Frze, Valung Indvdual Dfferences wthn Learnng: From Face-to-Face to Onlne Experence. Internatonal Journal of Teachng and Learnng n Hgher Educaton. 17(1), pp. 33-41, 2005. [9] A. Jeong, The Combned Effects of Response Tme and Message Content on Growth Patterns of Dscusson Threads n Computer- Supported Collaboratve Argumentaton. Journal of Dstance Educaton. 19(1), pp. 36-53, 2004. [10] C. Reffay, and T. Chaner, Socal Network Analyss Used for Modellng Collaboraton n Dstance Learnng Groups. In: Proceedng of Intellgent Tutorng System conference (ITS'02). Jun, France, pp. 31-40, 2002. [11] A. Martínez, Y. Dmtrads, B. Ruba, E. Gómez, and P. De la Fuente, Combnng qualtatve and socal network analyss for the study of socal aspects of collaboratve learnng. Computers and Educaton. 41(4), pp. 353-368, 2003. [12] R. Mazza, and C. Mlan, Explorng Usage Analyss n Learnng Systems: Ganng Insghts from Vsualzatons. In: Proceedngs of the 12th Internatonal Conference on Artfcal Intellgence n Educaton (AIED). Amsterdam, 2005. [13] P. Jerman, A. Soller, and M. Muhlenbrock, From Mrrorng to Gudng: A Revew of State of the Art Technology for Supportng Collaboratve Learnng. In: Proceedngs of EuroCSCL. Maastrcht, NL, pp. 324-331, 2001. [14] J. Zumbach, M. Muehlenbrock, M. Jansen, P. Remann, and U. Hoppe, Mult-dmensonal trackng n vrtual learnng teams: An exploratory study. In: Proceedngs of the Conference on Computer Supported Collaboratve Learnng CSCL-2002. Boulder, CO, pp. 650-651, 2002. [15] R. Bentley, W. Appelt, U. Busbach, E. Hnrchs, D. Kerr, S. Skkel, J. Trevor, and G. Woetzel, Basc Support for Cooperatve Work on the World Wde Web. Internatonal Journal of Human-Computer Studes. 46(6), pp. 827-846, 1997. [16] G. Stahl, Group Cognton: Computer Support for Buldng Collaboratve Knowledge. Actng wth Technology Seres. MIT Press, Cambrdge, MA, 2006. [17] C. Albrght, VBA for Modelers: Developng Decson Support Systems Usng Mcrosoft Excel. Duxbury Press, 2006. [18] T. Zapawa, Excel Advanced Report Development. Wley, New York, NY, 2005. [19] K. Berk, and P. Carey, Data Analyss wth Mcrosoft Excel. Duxbury Press, 2000. [20] D. Montgomery, and G. Runger, Appled Statstcs and Probablty for Engneers. John Wley & Sons, New York, NY, 2006. REFERENCES [1] S. Seufert, U. Lechner, and K. Stanoevska., A Reference Model for Onlne Learnng Communtes. Internatonal Journal on E-Learnng. 1(1), pp. 43-54, 2002. [2] M. Smonson, S. Smaldno, M. Albrght, and S. Zvacek, Teachng and Learnng at a Dstance. Prentce Hall, Upper Saddle Rver, NJ, 2003. [3] J. Engelbrecht, and A. Hardng, Teachng undergraduate mathematcs on the Internet. Part 1: Technologes and taxonomy. Educatonal Studes n Mathematcs. 58(2), pp. 235-252, 2005. [4] R. Sweet, Student Drop-out n Dstance Educaton: An Applcaton of Tnto's Model. Dstance Educaton. 7, pp. 201-213, 1986. [5] P. Dllenbourg (ed.), Collaboratve Learnng. Cogntve and Computatonal Approaches. Elsever Scence, 1999. [6] T. Daradoums, A. Martínez, and F. Xhafa, A Layered Framework for Evaluatng Onlne Collaboratve Learnng Interactons. Internatonal Journal of Human-Computer Studes. 64(7), pp. 622-635, 2006. [7] L. Guerrero, M. Madaraga, C. Collazos, J. Pno, and S. Ochoa, Collaboraton for Learnng Language Sklls. In: Proceedngs of 11th