STUDENT PARTICIPATION INDEX: STUDENT ASSESSMENT IN ONLINE COURSES

STUDENT PARTICIPATION INDEX: STUDENT ASSESSMENT IN ONLINE COURSES Alan Y K Chan, K O Chow Department of Computer Science City University of Hong Kong K S Cheung School of Continuing Education Hong Kong Baptist University ABSTRACT Online courses have been widely used to support teaching and learning in higher education. It is essential to develop methods to properly assess students in online course usages. This paper proposes a student participation index for assessing students in online courses. The index consists of a number of components, such as pages viewed and forum questions read and posted, and associated weights. It has the benefits of being non-course-specific, non-subjective, extendible and flexible. The development of the index and how it is used to evaluate students are described in the paper. Results indicate that students with higher index usually achieve better grades and vice versa. KEY WORDS E-Learning; Online Courses; Student Participation and Assessment 1. INTRODUCTION In recent years the proliferation of the Internet and the advancement of its technologies have a significant impact on the adaptation of e-learning in higher education. Online courses have now become an important component to support teaching and learning in classroom and are widely adopted today as part of the E-Learning initiative in higher education. They are usually delivered on learning management system, such as WebCT and BlackBoard. Online course evaluation is essential in order to improve the quality of teaching and learning. There are several ways to evaluate online courses, such as teacher/student reflection (questionnaires), student performance in assessments (assignments, quiz and examination) and student actions (web server log files and databases). Understanding student participation in online courses allows instructor to evaluate and assess students. The method for assessing student online participation should be flexible and extensible to allow for enhancement, and should not be based on subjective judgments. This paper proposes a student participation index (SPI) for assessing student in online courses. The index is computed based on student actions, based on what students do in online courses. Data are captured either in a form of web access log files or databases. 1

The next section discusses the characteristics and functions of online courses and criteria for evaluation. Section 3 describes the methodologies for developing the student participation index. Section 4 presents the result and how it is used to evaluate students. Finally, evaluation of the index and possible improvements are discussed in the conclusion. 2. STUDENT PARTICIPATION To understand and define student participation in online courses, it is essential to identify the characteristics and functions of online courses. These are discussed in the following sub-sections. 2.1 Characteristics of online courses Online courses provide a flexible learning environment where both instructors and students can teach and learn regardless of time and location. The learning process can be self-paced, independent, collaborative and continuous in online courses. Online courses support high quality learning by offering different kinds of environments such as synchronous or asynchronous or both. Some environments and approaches facilitates student learning while others impede it [1]. A traditional classroom provides a face-to-face learning environment where instructors can direct and take active immediate role in class at a fixed time and place. Online courses provide alternative opportunities for current on-campus students to take classes that they could not take otherwise due to time conflicts with other courses or work [2]. In traditional classroom, it is often instructional where the instructors transfer the knowledge directly to the students. It does not reflect the students understanding of the knowledge effectively. However, the dynamic nature in the online courses supports better communication such as self-reflections and peer-to-peer reviews. This change in communication transforms the learning process where knowledge is constructed actively by cognition [3]. 2.2 Functions of online courses Learning management system is commonly used as a platform for the delivery of online courses. LMS is defined as a distinct, pedagogically meaningful and comprehensive system by which learners and faculty can participate in the learning and instructional process at anytime and any place [4]. Its functions can be categorized into several components such as content delivery (organizer and content pages, URLs, etc), communication and collaboration (chat, whiteboard, forum, mail, calendar, etc), assessment (quizzes, assignments, self-test, etc) and class administration (grade book, syllabus, etc) [5]. 2.3 Evaluation of student participation Class attendance and contribution may be considered as student actions which can be used to evaluate student participation in the traditional classroom. In online courses, student actions include accessing course materials, posting and reading discussion forums, taking online quizzes, interacting synchronously 2

in chat room and white board, sharing resources, etc. They are usually stored in either databases or data files and their transaction log are recorded automatically in web access log files in a specific format. 3. METHODOLOGY The student participation index is an aggregation of various student actions in the online courses. The computation of the index is based on the weight of each pre-defined student action and the median of the students index scores. Ranking students by the index can be used for grading and comparison purposes. Table 1 below summarizes the constitution and the formula of the student participation index. There are three steps in the development of the index, namely defining the index components and their weights, defining the index formula and ranking the index results. The details are discussed below. Table 1: Constitution and formula of the Student Participation Index (SPI) Student Actions Variable Weight Score Number of pages viewed A 10% Score (A) = 10% * A / Max (A) Number of forum questions read B 20% Score (B) = 20% * B / Max (B) Number of forum questions posted C 30% Score (C) = 30% * C / Max (C) Number of chat sessions participated D 10% Score (D) = 10% * D / Max (D) Number of chat message submitted E 30% Score (E) = 30% * E / Max (E) Total Score = Score (A) + Score (B) + Score (C) + Score (D) + Score (E) SPI = 100 * Total Score / Median Score 3.1 Defining index components and their weights As discussed in the previous section, there are different types of student actions in the online courses. In this particular example, the student participation index consists of selected actions such as number of pages viewed, forum question read and posted, chat sessions and chat messages from each individual student. These actions are extracted from the log data in the online course. Each action is assigned a percentage weight. This assigned weighting is based on the subjective importance determined by the instructor. It should be noted that, depending on the course nature, different actions and weights may be included. 3.2 Defining index formula Each student s score is the summation of the score of actions. Each action score is calculated based on the weight, the student value and the maximum value of the action, as shown in Table 1. The maximum value of the action is used to achieve relative scoring. The median can be found after computing all students scores. It is course-specific and can be used to compute the index. The student participation index for a student is the student score divided by the median and multiplied by 100. If a student s index equals to 100, then he/she is in the median position of the student participation. 3

3.3 Ranking index results After calculating the student participation index, students can then be ranked for grading and comparison purposes. Instructor can define a score for online participation and students can be graded according to their position in the index. 4. RESULTS An online course in Java programming has been used as a sample for calculating the student participation index. The course has 72 students and is conducted in the WebCT platform. Results are described and interpreted in the following sub-sections. 4.1 Students by total score and index Figure 1: Student Score Pages viewed Articles read Articles posted Chat sessions Chat messages 100.00 90.00 80.00 70.00 60.00 % 50.00 40.00 30.00 20.00 10.00 0.00 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 Student Figure 2: Student Participation Index 300.00 250.00 200.00 150.00 100.00 50.00 0.00 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 4

Figures 1 and 2 show the total score and the index of each student in descending order. The percentage of each student action constitutes the total score. 4.2 Sample calculations There are 72 students in the sample online course. The highest and the lowest score are 93.61% and 2.15% respectively. The median score is 38.23%. Table 2 below illustrates the calculation of the student participation index for several students. Table 2: Examples of Student Participation Index calculation Variable Student 1 Student 2 Student 3 Student 4 Student 5 A 1011 765 717 481 304 B 350 356 320 259 63 C 38 5 2 3 0 D 3 3 3 3 3 E 112 64 43 42 10 Score (A) 8.93% 6.76% 6.33% 4.25% 2.69% Score (B) 19.61% 19.94% 17.93% 14.51% 3.53% Score (C) 30% 3.95% 1.58% 2.37% 0% Score (D) 10% 10% 10% 10% 10% Score (E) 25.07% 14.10% 9.63% 9.40% 2.24% Total Score 93.61% 54.75% 45.47% 40.53% 18.45% SPI 244.85 143.21 118.92 106.01 48.27 Rank 1 10 20 30 69 Notes: Max (A) = 1132, Max (B) = 357, Max (C) = 38, Max (D) = 3, Max (E) = 134 4.3 Comparison by rank between student grade and student participation index Figure 3 below is a scatter diagram for the ranking of students by their grades and student participation index. 5

Figure 3: A diagram of student ranking by their grades and student participation index Rank (Grade) 80 40 0 0 40 Rank (SPI) 80 As shown in Table 3 below, student can be categorized into four groups, namely high grade/high SPI, high grade/low SPI, low grade/high SPI and low grade/low SPI. There are more students with high grade/high SPI and low grade/low SPI. Table 3: Number of students in each category of online course participation Lower Grade Higher Grade Higher SPI 12 27 Lower SPI 20 14 4.4 Assigning score for online participation Students are assigned a score out of five according to their index. This score can be used as part of the student assessment. The score distribution is summarized in table 4. Table 4: Score for Student Participation Index Score Rank Number of Students 5 1 10 10 4 11 25 15 3 26 50 25 2 51 69 19 1 70 73 4 6

4.5 Interpretation of results For an online course, the relationship between student grades and SPI can have the following interpretation. Case (i) - High correlation between student grade and SPI: There are more students with high grade/high SPI and low grade/low SPI. Accordingly, there are fewer students with high grade/low SPI and low grade/high SPI. This implies that student participation is a critical factor for better student grade. Case (ii) - Low correlation between student grade and SPI: The portions of students with high grade/high SPI and low grade/low SPI are not significant. This implies that student participation is not a critical factor for better student grade. Our online course sample belongs to Case (i), based on the results (the scatter diagram for the ranking of students by grades and SPI) shown in Figure 3. There are relatively more students with high grade/high SPI and low grade/low SPI, implying that student participation is a critical factor for better student grade in the online course. 4.6 Implication of results These results have implications on the effectiveness of the online course. For an on-line course resulting in a high correlation between student grade and SPI, according to the above interpretation, the functions of the on-line course would have positive effects on the student grades. This implies that the on-line course is effective in the sense that it contributes to improve the student performance. In contrast, for an on-line course resulting in a low correlation between student grade and SPI, the functions of the on-line course would not have positive effects on the student grades. This implies that the on-line course is not effective in the sense that it does not contribute to improve the student performance. However, it should be noted that course effectiveness need to be evaluated among many indicators. The results should be regarded as one useful reference indicator for course evaluation purposes. 5. CONCLUSION This paper proposes a student participation index for student assessment in online courses. Student participation is defined from student actions during the use of the online courses. These include web log data on the number of pages viewed, forum questions read and posted, chat sessions participated and submitted. The development of the index involves three steps, namely defining the index components and their weights, defining the index formula and ranking the index results. Students can be assessed by assigning grades based on index ranking. Results from the web log of a sample course indicate that students with higher index usually achieve better grades and vice versa. 7

The student participation index is non-course-specific, non-subjective, extendible and flexible. Firstly, the index is not limited to any specific course. It is because the use of median during the computation of the index allows the index to be applied to different courses, resulting in an easy articulation of the student participation. Secondly, the index scores are student actions that are gathered from the log data. They are data and not subjective judgments. Thirdly, the index can be extended by adding more student actions to the formula. And finally, it is flexible because the weighting of student actions in the index is defined by the instructor and can be adjusted to fit various situations. These four benefits make the index a useful tool in assessing students in online courses. Since the log data are collected in batch at the end of the semester, one possible improvement is to gather them regularly during the semester, e.g. weekly basis. Student participation can be evaluated and instructors can continuously assess their students in order to refine teaching and learning strategies. We intend to experiment with other courses in subjects different from programming. REFERENCE [1] Lieblein, E., Critical factors for successful delivery of online programs, The Internet and Higher Education, 3(3), 2000, 161-174. [2] Schifter, C., Teaching in the 21st Century, Internet and Higher Education, 1(4), 1999, 281-290. [3] Cheng, C.C., Construction and Evaluation of a Web-Based Learning Portfolio System: An Electronic Assessment Tool, Innovations in Education and Teaching International, 38(2), 2001, 144-155. [4] Dringus, L., & Terell, S., The framework for directed online learning environments, The Internet and Higher Education, 2(1), 1999, 55-67. [5] Coldwell, J., Mapping Pedagogy to Technology A Simple Model, In Proceedings of the 2nd International Conference on Advances in Web-Based Learning, 180-192, 2003. 8