Research on the strategies of improving Online Testing Credibility

Transcription

1 Research on the strategies of improving Online Testing Credibility Ruey-Shin Chen * National Quemoy University rueyshin_chen@nqu.edu.tw 摘 要 為 了 增 加 線 上 考 試 的 可 靠 性, 本 研 究 利 用 命 題 的 策 略 或 技 巧 來 提 高 線 上 考 試 的 有 效 性, 其 中 也 應 用 一 個 可 靠 的 網 上 考 試 政 策 以 及 先 進 的 臉 部 識 別 記 錄 系 統 來 完 成, 在 調 查 的 551 名 學 生 中,90% 的 網 上 學 習 的 學 生 認 為, 線 上 考 試 的 分 數 是 可 信 的 根 據 前 測 與 後 測 的 數 據 分 析, 所 有 的 p- 值 都 有 顯 著 的 差 異 性, 表 示 本 次 所 採 行 的 網 上 考 試 政 策 以 及 先 進 的 臉 部 識 別 記 錄 系 統 是 有 效 的 提 升 考 試 的 成 績 可 信 度, 但 大 部 分 的 平 均 值 分 佈 不 夠 高 經 過 數 據 分 析, 事 後 我 們 採 訪 了 36 名 學 生, 獲 得 了 一 些 改 進 線 上 考 試 的 方 向, 可 以 供 學 校 的 教 師 或 行 政 人 員 作 為 政 策 的 參 考 關 鍵 詞 :Online education, Online Testing Abstract In order to increase Online Testing reliability, this paper aims to contribute to improving Online Testing effectiveness and developing a credible online exam policy through the application of question skill and advanced face identification and recording technology. In a survey of five hundred and fifty-one students, ninety percent of the online study students regard that Online Testing scores are credible. According to data analysis, all p-values of questionnaires are significantly different between pretest and posttest, but the distributions of all posttest Means are not high enough. Then, after data analysis, we interview 36 students and obtained some improvements of Online Testing Policy for future reference of school teacher or faculty. Keywords: Online education, Online Testing Introduction Due to the advancement in information technology, online exam pattern has changed the traditional practice of examination. Teacher and students attitude on the credibility and reliability of the online exam is a factor to the success of online exam. The reason is that educational researchers have examined the enormous diversity of behaviours and issues involved in cheating of examination (Crown & Spiller, 1998). How Teachers use the optimal management strategy with existing information technology to enhance the credibility of the online exam to promote the effectiveness of online learning and develop students' independent learning and self-responsible behaviour is required deliberative teachers during the design of online exam. Learner-centered strategies are employed in this study, we adopt diversified manageably exam execution strategy and give examinees detail exam explanation. Then, we adopt existing information technology on online test. After students finish online test, they were asked to finish questionnaires of investigation. According to the student s credibility on the online exams, teachers can improve their strategies for the future online exam implementation. This research discusses issues of the existing factor which affect the credibility of Online Testing and explore existing online exam strategy to enhance students' satisfaction and learning effectiveness. The purpose of this study includes: 1. To explore existing information technology and strategies implemented on ONLINE TESTING for after-school exam and evaluate whether students recognize the fairness of the test scores. 2. Explore how ONLINE TESTING influence student s learning behavior. 3. To assess whether strategies used by the teacher can enhance student learning initiative and confidence Literature Review Notarization of Online Testing Online Testing based on whether it is implemented by the invigilators; the most uncontroversial condition is only used for learner self-assessment. Recent research has also noted that current web-based exam systems are preferable to traditional exam systems for conducting self-assessments (Wang, 2011). In the current situation, if teachers want employ Online Testing results as the basis of student learning assessment. There is some controversy especially when Students perform Online Testing with no supervision because the quality of education has been a continuing concern for faculty and administrators in higher education. A study of the frequency of cheating on college campuses found that a majority of students still cheats on some exams (McCabe, 1992; Bolin, 2004). It is difficult for testers to monitor credible assessment in a web-based exam system due to insufficient student identification. According to a study of online exams, 45% of students reported negative behaviours, such as lying, cheating, and defiance, during online exams (Russell & James, 2008). As some online education system employed by other researchers, in this study, we use Notarization functions of ecampus system which is developed by Taiwan 3Probe Technologies and Quemoy 32

2 University. This learning and examination system can be selected from the options by using the computer with an embedded webcam screen. It can require students to provide Notebook or computer's MAC address registration and web links when Students in off-campus study or examination. This feature can also record students' portfolios and portrait as a process tracing of student s online learning. Effectiveness of Online Learning and Testing System Most online education system with a common learning environment which provides interaction between students or teachers and students, to provide students with self-mind to explore creative activities and to stimulate and maintain a high learning interactive. The purpose of the examination is to assess student s learning outcomes or effectiveness for teaching. According the level of scores, students read it as diligent assessment for their self-adjustment reference. This procedure helps students to construct their own concept of knowledge between learning and examination. The best online teaching and examination system should be the teaching system in the face of a different course, different levels of students able to use different modes of teaching and examination to accept counselling. Teachers can also understand students' learning outcomes from student learning and examination process records. According to some researchers reports, most e-learning, distance learning or online learning system, are showing their effectiveness on student s knowledge acceptation, perception and learning attitude (Dimitriadis, et al, 207; Looi, Lin, & Liu, 2008; Chen, Looi, & Tan, 2010; Chen & Looi, 2011). By the portrait recording mode on the online learning and examination process, it enables students to achieve self-vigilance and self-supervision in online learning or examination function. However, it still requires clever planning of the teacher in the online course and online exam strategy. Online Testing Policy The development of Online Testing strategies affects the credibility of test score and Online learning performance. A good policy should have good coordination and implementation among education, science and technology (Stahl et al., 2006). ONLINE TESTING has been widely used in many online educations because it provides a convenient alternative assessment tool for evaluating students online learning capability. The popularity of online Testing is due to the convenience and ease of use of the online exam site. However, in the current educational system, there is a considerable gap of reliability between Online Testing and traditional exams. Most students who take online exams tend to believe that some degree of cheating is acceptable for online exams. Thus, cheating behaviours, such as substituting examinees or discussion with others at test, are likely to occur. Implementing some policy such as a notarisation process that includes face recording and verification by authorised educational staff upon an examinee's initial registration with the school, such as the proposed e-exam system, can effectively improve trust in the examinee and e-test score validity. Therefore, test score reliability and validity have been recognised as the most important factors in a successful online exam site. Given the convenience of the ONLINE TESTINGs, governments and educators are challenged with New Policy and establishing fair environments for electronic exams. As the need for Online Testing continues to grow, teachers or faculties need employ new policies such as more complete student identification, including a face recording system for examinees (Chen, 2008). Therefore, this paper aims to contribute to improving Online Testing effectiveness and developing a credible online exam policy through the application of question skill and advanced face identification and recording technology for examinees. Research Method Study Object The objects of study are students in Taipei and Kinmen County University who are enrolled in Information related courses. They are 392 students participating in the pretest and 159 students in the posttest in a total of 551 students. They use the tool with portrait record function for their Online Testing. The issue heterogeneity of pretest and posttest analysis is employed for the data collected in this research. Research Design In this study, based on the theory of Policy and extend TAM (Chen & Liu, 2013), the test content is designed by referencing the information relevant courses of the University. Before Online Test Stage questionnaires were used to explore the score credibility of Online Testing from students. Online Education Stage Students perform Online Education and understand online examination policy; and Online Testing and Questionnaires Students follow the Online Testing policy and finish the examination and questionnaires. The arrangement of experimental flow is as the following procedure: the first week is active preparations. We employ pretest questionnaires collection and the inspection of online testing system; Second to third week, we employ Online Education activity. During online education period, teacher advocates his Question Policy and ask students enroll Online Testing and upload their Portrait. Then, the teaching aid proofs student s identification. The fourth week, we proceed Online Testing and collect posttest questionnaires; the fifth 33

3 week, we proceed Interview activities, data aggregation and data analysis. There are120 PCs with embedded webcam monitor involved in our research. ecampus also provides students uploading their portrait and perform Online Testing registration. In this stage, Teaching Aid performs student s identification confirmation and advocate teacher s testing Policy; ID confirmed students can start his Online Education. The can do it on Internet at home or through wireless at University Campus. The Policy of this Online Testing experiment is the total Online Testing time is 30 minutes and fixed in certain period. The total number of question will be appropriate. Students will finished the testing just in time and have no time to discuss each other. The sequences of question are randomly select from questions base by the Online Testing system. The student will never have the same answer sequence even they sit side by side. Before online testing, testee need to agree ecampus System capture and record his big head pictures, Students can take their examination at home with qualified webcam system. Research Tool and Teacher Policy The operation tools include ecampus learning and testing platform, tablet PCs, and teacher self-made material. The research tools include display screen with embedded webcam, digital test paper, questionnaires and SPSS statistic software. The experiment of this study also employs Online Testing tool with the function of portrait recording system which is jointly developed by Quemoy University and 3Probe Technologies. For teacher s Policy options, it provides portrait registration before examination by retrieving the portrait and recording it during the exam period. The process is simple. Transcripts can be tied to registered portrait and three random captured portraits for ID comparison. In order to reduce the chance of cheating occurred, teacher can specify the question orders which randomly selected from the question base. Teacher also can define common examination period for preventing of cheating. Research data are collected from online questionnaires system after student finish ecampus Online Testing operation. The same questionnaires are adopted at pretest and posttest. Questionnaires are edited from Extending the TAM model to explore the factors that affect Intention to use an online learning community (Liu, et al., 2010) which is suitable for student s comprehension and expression. In this research, questionnaires scores on Likert five-point scale. The 5 factors include Participation, Notarization, Testing System, Teacher Policy, and Credibility & Effectiveness. After data analysis we interview some student in order to explore students actions in our questionnaire answers. Result Analysis All examinees have e-learning experience according to students background check. 94% of students will take the initiative to choose the course of online learning and examinations. 89% of students are sure that the function of the portrait records used at online learning and exam activity. 91% of students are sure that the function of the portrait records used at online learning which can promote Learning Effectiveness. 93% of students are sure that the function of the portrait records used at Online Testing which is one of the effective methods to prevent cheating test. 95% of students hope school provides more online learning and exam courses. 96% of students hope school creates more complete online examination system and process flows. 90% of students are sure that the function of the portrait records used at Online Testing which promote the credibility of testing scores. 93% of students are sure that Online Exam system is reliable. The following are results of data analysis employed by SPSS statistic software on questionnaires: The analysis of Pretest and Posttest They are 392 students in the pretest (questionnaires before using Portrait Recording function and Teacher s new Policy in Online Testing) and 159 students in the posttest (after using new Policy). An independent-samples t-test was conducted to compare research issues for two groups of students. According to the Compare Mean t-test of two Independent Samples, the resulting reports are listed below: (also regarded as 6 issues including Participation, Notarization, Testing System, Teacher Policy, and Credibility & Effectiveness) Issue 1-Participation. Students will take the initiative to choose the course of online learning and examinations. There was a significant difference in the Issue 1 for Before New Policy (M = 3.69, SD = 0.043) and After New Policy (M = 3.96, SD = 0.69) conditions; t (549) = , p =.001 (two-tailed). These results suggest that New Policy really does have an effect on that Students will take the initiative to choose the course of online learning and examinations. Specifically, our results suggest that when teacher employs new online testing Policy, students will take the more initiative of Online Testing or Learning. Issue 2-Teacher Policy. Students hope school creates more complete online examination policy and process flows. There was a significant difference in the Issue 2 for Before New Policy (M = 3.88, SD = 0.042) and After New Policy (M = 4.06, SD = 0.71) conditions; t (545) = -2.24, p =.028 (two-tailed). These results those Students hope school creates more complete online examination policy and process flows. 34

4 Specifically (M=4.06), our results suggest that students continually wish teacher employs more complete online testing or process Policy. Issue 3-Notarization. Students are sure that the function of the portrait records used at online learning which can promote Learning Effectiveness. There was a significant difference in the Issue 1 for Before New Policy (M = 3.53, SD = 0.464) and After New Policy (M = 3.94, SD = 0.75) conditions; t (547) = -4.66, p =.000 (two-tailed). These results those Students are sure that the function of the portrait records used at online learning which can promote Learning Effectiveness. Specifically, our results suggest that when teacher employs new online testing Policy, students agree it can promote their Learning Effectiveness. Issue 4-Testing System of portrait recording. Students are sure that the function of the portrait records used at Online Testing which is one of the effective methods to prevent cheating test. There was a significant difference in the Issue 1 for Before New Policy (M = 3.74, SD = 0.047) and After New Policy (M = 4.03, SD = 0.07) conditions; t (546) = , p =.001 (two-tailed). These results suggest that New Policy really does have an effect on those Students agree that the function of the portrait records used at Online Testing which is one of the effective methods to prevent cheating test. Specifically, our results suggest that when teacher employs new online testing Policy, students will have less cheating on Online Testing or Learning. Issue 5-Credibility. Students are sure that the New Policy of Online Testing which increases the credibility of testing scores. There was a significant difference in the Issue 1 for Before New Policy (M = 3.51, SD = 0.044) and After New Policy (M = 3.94, SD = 0.073) conditions; t (541) = -5.03, p =.00 (two-tailed). These results Students credibility of score accepted form online examinations. Specifically, our results suggest that when teacher employs new online testing Policy, students will have more score credibility on Online Testing. Issue 6-Effectiveness of Satisfaction. Students will satisfy portrait recording on online learning and examinations. There was a significant difference in the Issue 1 for Before New Policy (M = 3.49, SD = 0.049) and After New Policy (M = 4.03, SD = 0.71) conditions; t (547) = , p =.000 (two-tailed). These results suggest that Portrait recording really does have an effect on that Students satisfaction in online examinations. Specifically, our results suggest that when teacher employs new online testing Policy, students will have score credibility accept from Online Testing. Data analysis and Discussion According to Table 1, there is a significant difference result from the questionnaire 1to 5 for before and after New Policy conditions; all 5 p-values are less than.01 (two-tailed). These results Students willing of taking the participation to online learning and examinations. Specifically, our results suggest that students will take the more initiative of using Online Testing or Learning when a teacher employs new online testing Policy. Students are willing to use more computer networks as the online exam tools and feel more convenient in online exam process. They are more satisfied with the online exam software and hardware facilities and participate more in the online exam again. From the Table 1, 5 p-values are significantly less than.001 but the distribution of 5 Means are from 3.90 to These Means are not higher than 4.5. It shows that New Online Testing Policy is significant but there are some insufficient parts for students view point on participation. Table 1 Q Q Q Q Q Q1. I will take the initiative to enjoy in online learning and examination courses Q2. I will use a computer network as the online exam tools Q3. I feel quite convenient in online exam process Q4. I am satisfied with the online exam software and hardware facilities Q5. Overall, I am willing to participate in the online exam again According to Table 2, there was a significant difference in questionnaire 6 to 10 for before and After Face Notarization conditions; all of 5 p-values are less than.01 (two-tailed). These results suggest that Face Notarization really does have an effect function on Students confirmation of online learning and examinations. Specifically, our results suggest that when teacher employs Face Notarization on online testing Policy, students will be surer the online exam portrait recording functions. From the Table2, 5 p-values are significantly less than.01 but the distribution of 5 Means are from 3.84 to These Means are not high than 4.5. It shows that New Face Notarization Policy is significant for Online Testing but there are some insufficient parts for students view point On Face Notarization. 35

5 Table 2 Q Q Q Q Q Q6. I am sure that online learning and exam portrait record Q7. online exam irregular Portrait recording of testees is one of the effective methods Q8. online learning irregular portrait record students identity, which enhance effective learning Q9. through online portrait recording function to learn, I will be not indolent in learning Q10.through online portrait recording function, I would be more attentive to learning According to Table 3, there was a significant difference in questionnaire 11 to 14 for before and after School Policy; all of 5 p-values are less than.05 (two-tailed). These results suggest that Face Notarization really does have an impact on Students understanding of School Policy. Specifically, our results suggest that when teacher employs New School Policy on online testing Policy, students will be more aware of the School Policy. From the Table3, 5 p-values are significantly less than.05 but the distribution of 5 Means are from 4.01 to These Means are all higher than 4.0. It shows that New School Policy is significant for Online Testing, but there are anticipating that school provide more good policy or chance for students participating in online learning and exam courses. Table 3 Q Q Q Q Q11.I hope schools provide more chance for students participating in online learning and exam courses Q12.I hope schools encourage more students to participate in online learning and exam courses Q13.I hope school perfect the online exam digital environment Q14.I hope school establish a more complete online examination system and process According to Table 4, there was a significant difference in questionnaire 15 to 18 for before and After School Policy conditions; all of 5 p-values are less than.01 (two-tailed). These results suggest that New Policy really does have an effect function on Students satisfaction and credibility of online learning and examinations. Specifically, our results suggest that when teacher employs on online testing Policy, students will be surer the satisfaction and credibility of Online Testing. From the Table4, 4 p-values are significantly less than.01 but the distribution of 4 Means are from 3.91 to These Means are not high than 4.5. It shows that School Policy is significant for the satisfaction and credibility of Online Testing but there are some insufficient parts for students view point on credibility. Table 4 Q Q Q Q Q15.I will be satisfied with the online examination with portrait recording function Q16.Portrait record can enhance the credibility of the results of an online course Q17.the online exam security mechanism is reliable Q18.online exam system is creditable Conclusion According to data analysis, all p-values of questionnaires are significantly different between pretest and posttest. Notarization, Testing System, and Teacher Policy will influence Credibility & Effectiveness of online testing, but the distributions of all posttest Means are around 4. These Means are not higher than 4.5. It shows that School Policy is significant for all questionnaire issues implementing at Online Testing, but this study needs to explore the reasons why students still show some insufficient feelings for this new policy. After data analysis, we interview 36 students and obtained some improvements of Online Testing Policy for future reference of school teacher or faculty as following. 1. There is no further opportunity to take examination online after the due time when the teacher limits the exam time period. Teacher should consider the connection reliability of internet or wireless network. 2. Students need to have face to face interaction with other classmates so the e-learning and online testing may not replace all traditional styles. 3. Online Testing style only can be the style of Multiple choices, Short-answer, Single choice. Writing test is currently not sufficient enough to be scored by Online Testing. 4. In order to improve the credibility of this Testing 36

6 Policy, voice recording function need to be included in future Online Testing System. 5. Timed testing questions can also be separately administered for each question to prevent students from searching for unmonitored help whether from peers or online materials. 6. For the first time use students, they hope teacher provide chance of practice before the real online examination. Reference: [1] Bolin, A. (2004). Self-control, perceived opportunity, and attitudes as predictors of academic dishonesty. Journal of Psychology, 138, Russell, S. M., James, L. R. (2008). Recording lying,cheating, and defiance in an Internet Based Simulated Environmen. Computers in Human Behavior, 24(5), [2] Chen, R. S., Hsieh, K. K., & Tsai, C. H. (2008). The implementation of face recognition technology and its effect on e-quiz credibility. Asian Journal on Quality, 11(2), [3] Chen, R. S., & Liu, I. F. (2013). Research on the effectiveness of information technology in reducing the Rural Urban Knowledge Divide. Computers & Education, 63, [4] Chen, W., Looi, C. K., & Tan, S. (2010). What do students do in a F2F Online Testing classroom? The optimization of multiple communications modes. Computers & Education,55(3), [5] Chen, W., & Looi, C. K. (2011). Active classroom participation in a Group Scribblesprimary science classroom. British Journal of Educational Technology, 42(4), [6] Crown, D. F., & Spiller, M. S. (1998). Learning from the literature on collegiate cheating: A review of the empirical literature. Journal of Business Ethics, 17, [7] Dimitriadis, Y., Asensio, J. I., Hernandez, D., Roschelle, J., Brecht, J., Tatar, D. et al. (2007,July). From socially-mediated to technology-mediated coordination: A study of design tensions using Group Scribbles. Paper presented at the Computer Supported Collaborative Learning 2007 Conference, New Brunswick, NJ. Johnson, D. W., Johnson, R. T., & Stanne, M. B. (2000). Cooperative learning methods: A meta-analysis. Retrieved December 14, 2009, from [8] Liu, I. F., Chen, M. C., Sun, Y.S., Wible, D., & Kuo, C. H. (2010). Extending the TAM model to explore the factors that affect Intention to use an online learning community. Computers & Education, 54(2), [9] Looi, C. K., Lin, C.-P., & Liu, K.-P. (2008). Group Scribbles to support knowledge building in jigsaw method. IEEE Transactions on Learning Technologies, 1(3), [10] McCabe, D. L. (1992). The influence of situational ethics on cheating among college students. Sociological Inquiry, 62(3), [11] Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp ). Cambrige, UK: Cambridge University Press. [12] Skemp, R. R. (1989). Mathematics in the primary school. London: Routledge. [13] Wang, T. H. (2011). Implementation of Web-based dynamic assessment in facilitating junior high school students to learn mathematics. Computers & Education, 56(4),