Contrast and validation of computer assisted language testing in Chinese students learning Spanish

Transcription

1 Contrast and validation of computer assisted language testing in Chinese students learning Spanish J. M. de Siqueira 1, D. González 1, J. García Laborda 1, and T. Magal Royo 2 1 Department of Applied Linguistics, Polytechnic University of Valencia, Camino de Vera s/n, Valencia, Spain 2 CITG, Polytechnic University of Valencia, Camino de Vera s/n, Valencia, Spain This study shows the technical as well as the technological and linguistic aspects needed to develop exam platforms for projects as Selector (sponsored by the Government of the Valencian Region), Ingenio (sponsored by the Polytechnic University of Valencia), CALL@C&S (sponsored by the European Union), and Paulex Universitas (sponsored by the Spanish Ministry of Science and Innovation). Examples will be given from these projects as a result of more than 8 years of investigation in the matter. The main goal is to contrast and validate some of these aspects in students with specific needs and constraints, for instance students learning a language with an alphabet other than that of his mother tongue. For the purposes of this survey an experiment has been done at the Polytechnic University of Valencia with 48 Chinese students that have come to Spain to study Spanish. The results of various exams done by these students have been compared; those corresponding to the intensive Spanish course and other two exams corresponding to the preparation for the official exam DELE (Spanish Foreign Language Certificate) both on paper and with computer assistance. The results of this survey show the difficulties experienced by Chinese students in the use of a Spanish keyboard, the insecurity felt by these students when dealing with a computer exam and their preferences expressed through a specific questionnaire comparing the test in paper format with the same test in the computer version. The study concludes that all these technologic aspects need to be analyzed thoroughly from the different cultural and linguistic perspectives of worldwide students. Keywords e-learning; Assessment; Computer Assisted Language Testing; Chinese Students; Usability 1. Introduction The development of platforms for computer assisted language testing has a long history. From the first multiple choice platforms of the 70 s to platforms combining tasks such as TOEFL there have been changes that are now rarely considered by linguists. However technical specifications give shape and sense to the exam construct. For instance it is necessary to consider the specifications of the TOEFL platform to understand how tasks are and what the effect is on language learners. Many prior surveys have justified this connection between construct and design of testing platforms in matters like the use of images and audiovisual repertoires [1-2], ergonomics [3-4], computer familiarity [5] and context [6]. All these factors have to be defined clearly so that the assessment is valid and provides the highest reliability required for the linguistic assessment. This study shows the technical as well as the technological and linguistic aspects needed to develop exam platforms for projects as Selector (sponsored by the Government of the Valencian Region), Ingenio (sponsored by the Polytechnic University of Valencia), CALL@C&S (sponsored by the European Union), and Paulex Universitas (sponsored the by Spanish Ministry of Science and Innovation). Examples will be given from these projects as a result of more than 8 years of investigation in the matter. The main goal is to contrast and validate some of these aspects in students with specific needs and constraints, for instance students learning a language with an alphabet other than that of his mother tongue. For the purposes of this survey an experiment has been done at the Polytechnic University of Valencia with 48 Chinese students that have come to Spain to study Spanish. The results of various exams done by these students have been compared; those corresponding to the intensive Spanish course and other two exams corresponding to the preparation for the official exam DELE (Spanish Foreign Language Certificate) both on paper and with computer assistance. The results of this survey show the difficulties experienced by Chinese students in the use of a Spanish keyboard, the insecurity felt by these students when dealing with a computer exam and their preferences expressed through a specific questionnaire comparing the test in paper format with the same test in the computer version. The study concludes that all these technologic aspects need to be analyzed thoroughly from the different cultural and linguistic perspectives of worldwide students. 283

2 2. Method The study was conducted at the Polytechnic University of Valencia, Spain, as one experiment framed in the PAULEX Project linked to Computer-Assisted Language Testing (CALT). As a qualitative method, five Spanish teachers have been previously interviewed in order to compile information about their experiences and beliefs concerning the significant features about using Computer-Based Tests (CBT) with Chinese students learning Spanish. Then, the findings that could influence Chinese students doing Spanish CBTs have been considered to this experiment and analysed through two different input mechanisms: a validated questionnaire compiling personal preferences and the results of tests done by students Participants The Spanish intensive courses for Chinese students held at the Polytechnic University of Valencia derives from a Cultural, Educative and Scientific Cooperation Agreement signed between Spain and the Republic of China in Beijing on the 21st October 2007 with the aim to consolidate cultural and educative relationships and to promote cooperation and exchange between both countries [Spanish Official State Gazette, BOE]. By virtue of this Agreement Chinese students are granted the recognition of their Gao Kao (examination to access university studies) so that Chinese students can make their college studies in Spain and, viceversa, Spanish students see their specifics exams recognized to study a degree in a Chinese university. The Agreement also includes the future recognition of university degrees and official institutions certificates of both countries subject to their own legislation and education system. Following this Agreement, a delegation of the Polytechnic University of Valencia flew to China to offer some Spanish courses to Chinese Gao Kao students. A group of 48 are currently having Spanish lessons at the Gandía Campus of the Polytechnic University of Valencia since October Students come from several provinces of China. All of them are between 17 and 21 and have studied Spanish in different schools. A 500-hour course of Spanish language was made at the same time that the Gao Kao exams were being prepared and so students had to do a major effort to combine both studies. Teaching Spanish to Chinese students constitutes a major challenge for the group of teachers involved in these courses because of the interlinguistic difference [7] of the Spanish and Chinese language. Communication problems arise frequently- especially at the first stages of the course- due to the differences between teacher and students when codifying and decodifying linguistic elements, using pragmatic rules and approaching cultural and ideological issues. The Chinese and the Spanish language are situated at opposite ends of the interlinguistic sequence and this becomes clear when approaching syntactic, phonologic or morphologic aspects. Similarly, difficulties emerge when students need to deal with semantic, stylistic and pragmatic and body-language aspects showing problems to consolidate the contents learnt. All these key factors determine the methodology and the materials produced and used in class. As for the students, the challenge they have to face is even bigger, as they need to integrate themselves in new cultural and society patterns. Initially they found it complicated to establish personal relationships in the Campus with Spanish and other foreign students as they could not communicate fluently in Spanish and so the Chinese students grouped together and did not enjoy the life of the Campus. However, students have recently started to enrol in several cultural and sports activities which are helping them to integrate in the life of the Campus. As soon as the students started introducing new habits and activities with Spanish students, their Spanish boosted enormously. We should also bear in mind the situational frame in which Chinese students are learning Spanish and that social conditions and continued input influence the opportunities that learners have to hear and speak the language [8]. It is worth mentioning the role that English as Second Language (ESL) plays in our Chinese students. Most of the students studied English in China and have arrived in Spain with a pre-intermediate level. Interlanguage is viewed by the Chinese students as a valuable tool for the purposes of learning and they make use of it specially in semantic and syntactic aspects -Spanish and English have more than 60% of their lexicon coming from a greco-latin origin. In this case, although the interference may be present, the advantages are more numerous than the drawbacks: the student is aware of the linguistic similarity of both English and Spanish and makes hypothesis to transfer his knowledge of ESL. In initial stages, with certain students that have no Spanish previous instruction, English can be a valuable tool both for the teacher to facilitate communication and save time as well as for the student to feel confident on his linguistic resources. The fact that students have learnt English also influences them to use technology. Nearly all the students came from China with brand-new laptops and dictionary gadgets to help them with the Spanish language. When they were asked about the possibility of computer-based tests not a single student rejected the proposition. 284

3 2.2. The questionnaire The questionnaire was developed to provide statistically significant findings using quantitative analyses. It lasted about 5 minutes and consisted of 25 multiple-choice questions. It was generated with the intention not to be exhaustive, otherwise the experiment would have taken an excessive time to be completed. The questions were divided into three sections: Personal Characteristics; Computer Experiences and Anxiety; and, comparing beliefs, preferences and feelings about CBT and P&P. The first section, with 4 questions, compiles personal characteristics to classify participants into different groups in order to identify parameters which could influence the questionnaire results. The second section, with 6 questions, compiles information about Computer Anxiety and Computer Experience taking into account several studies and validated scales [9-11]. The last section, with 15 questions, recognizes information about preferences and feelings from participants when comparing CBT and P&P. All questions of the second section and 10 of the third section use Linkert scales The language test The tests were developed taking into account pedagogical parameters and psychometric tasks concerning language testing in order to make them similar and valid for the assessment of Spanish language students. Additionally, the final marks of students on the regular Spanish course of the University were considered to contrast and validate the results of tests of this experiment. Since these students are studying to get the certification of DELE, tests were prepared to be as similar as possible. DELE stands for Diploma de Español como Lengua Extranjera, or Spanish as a Foreign Language Diploma. It is the only certificate for non-native Spanish speakers that is officially recognized by the Spanish Ministry of Education, Culture and Sport. This diploma is of great international value for those who are going to study at a Spanish University and who want to work, or are going to work in a Spanish speaking country environment. It is granted by the Instituto Cervantes, a public institution created in 1991 to promote the Spanish and Latin American language and culture. Centres of the Institute have spread all over the four continents during the last years. There are three levels: Inicial (Beginner), Intermedio (Intermediate) and Superior (Proficiency) which are structured in accordance with the European Higher Education Area (EHEA) and the standards and contents established by the Instituto Cervantes for the studies of Spanish as Foreign Language [12]. Our Chinese students are preparing for the Intermediate DELE examination, which corresponds to level B2 of the European Frame of Reference. Learners must prove that they possess the necessary language skills to be able to take part spontaneously in routine communication held at a natural pace. The certificate shows that they can deal with all situations that are of interest to them in both private and professional contexts. The Spanish language exam for all levels consists of 5 different parts. The main difference of the 3 levels is the difficulty and the time available for each part. The first part examines the student's reading comprehension. In this part students read a text which will vary in length and difficulty depending on the exam they are taking. After reading the text the student has to answer a number of multiple choice questions about the text. In the second part of the DELE exam writing skills are tested. In this part students are asked to write a small text an essay depending on the Spanish level of your exam. The third part is made up of listening exercises where the student has to answer a series questions of different typology after hearing a Spanish text read aloud. The fourth part of the DELE Test consists of multiple choice questions and filling in blank spaces with the correct Spanish word. This part measures students knowledge of grammar and vocabulary. The oral test comes last. In this fifth part the student is interviewed and has to talk to the examiner in Spanish for about minutes describing a picture or explaining his viewpoint about a specific topic. Whereas the listening and the reading comprehension sections are seen as difficult but not impossible - goals to accomplish, students usually find the oral and the written sections of the exam extremely difficult. Therefore students have to work twice as hard to be able to pass these test sections. Special activities have been designed to reinforce these skills bearing in mind the tight lesson schedule and the closeness of the examination date (May, 15th). Concerning the CBT, the use of different multimedia and interactive elements was avoided in order to make both formats as similar as possible. Despite of all, excluding some of the possibilities offered by technology for CBT and making it similar to a P&P limited the results of the experiment by using just the first of the three generations of testing systems argued by Bennett [13]. He defines that the first generation takes limited advantage of technology, making it very similar to the P&P ones, the second includes new possibilities such as multimedia elements and the automatic item generation, and the last one consists of continuous assessment during the learning process. Currently there are Learning Management Systems (LMS), as the Ingenio System [14], which are consolidating their tools related to this third generation of testing systems. 285

4 2.4. Procedure The students, after reading brief instructions and asking for procedures, completed first the CBT, then the P&P and finally the questionnaire. The results were contrasted with the final marks of the regular Spanish course and no important deviation was found. While doing the CBT, the system was automatically saving on the server all the important information from the participants, including answers and the time spent on each section of the test. On P&P, students had to mark by themselves the time spent on each section. Once completed the questionnaire, a full report was prepared recompiling information from the CBT, the P&P and the questionnaire. All the language test answers were included in this report in order to confirm that participants had paid enough attention and had answered all questions satisfactorily. These reports, including all the test answers and specially the free writing section, have been used for other studies about students results of CBT and P&P. All information from questionnaires and tests was ported into a statistical analysis package program (SPSS) for a significant statistical analysis (level of 0.05). 3. Results Some significant features on comparing results of tests and the questionnaire were found. The main aim of this experiment was the identification of these features within the frame of the PAULEX Project. Nevertheless, the principal objective of this article has been the compilation of some especial issues particularly linked to the background of the Chinese students. The first important finding was that, using simple factorial ANOVAs analysis, some statistically significant differences and other unimportant ones were identified. All the statistical analyses were reconsidered by classifying participants into men and women in order to look for sex differences, but no significant differences were found. With regard to the participants experiences and usage of technology, the more experience in technology the participants had, the better results on CBT they got. All students preferred the CBT instead of P&P, considering the CBT as more appropriate (67%) and preferable (56%) at universities. They defended that are already prepared for the CBT, but they first have to simulate and practice because they almost didn t use it in schools in China. Concerning time, important differences were found when comparing the CBT and the P&P results. First, Chinese students needed more time to complete the exams. Students were unable to control time and devoted more time to reflect and to mark an answer in sections where they needed to provide a quick one (as in multiple choice questions). Also a portion of students did not finish the writing part or handed over only one essay, although the writing section includes two essays of words each in 60 min. Regarding readability, only four students out of all the participants preferred reading the text on a computer screen, and there are not significant differences in relation to the time spent on reading. No important time differences were found in the multiple-choice section, and 95% of participants reported to be indifferent about doing this section by means of CBT or P&P. Most of the important differences were recognized in the writing section. Some important deviations were identified in the case of some participants who had more difficulties when writing on computers since they are used to using a totally different alphabet. This preference, in comparison with the frequency of using computers for writing, had been proved in SPSS as a statistically significant feature (P<0.05). All participants who write more than 100 words daily or weekly preferred the CBT writing format, while 92% of those who seldom or sometimes use computers for writing in Spanish chose the P&P format. 4. Conclusions This paper attempts to establish clear guidelines for computer testing for the DELE and university internal examinations through a practical case with Chinese students. The paper suggests that it is important to develop a research agenda from two different but complementary perspectives: language testing theory and implementation, and software design. The university has contrasted the validity of the testing platform for a variety of purposes, especially for Spanish and English as a foreign language. Decisions on a future implementation of this project correspond to the Ministry of Science and Research as well as the Ministry of Education. However, tests are necessary in light of external academic needs and also to national working demand. Thus, the development of precise testing software, its trialing and implementation should be welcome. According to our observations, Chinese students use computers for their personal leisure. They are active consumers of technology and they use computers regularly to communicate with friends and family. However, only recently some of them have been able to use computers for pedagogical and learning reasons especially in the mother tongue. Since Chinese students had no previous experiences in CALT, during the CPT they faced problems such as the lack of time, their limited familiarity with the tasks and computer induced stressful 286

5 situations. Other problems included their reactions to face-to-face situations, changes in the alphabet, keyboard changes, lack of expertise, changes in the testing format in relation to other tests, emphasis on the final product and not on the processes, etc. The written test is the one posing more problems. The rest of the examination sections are not complicated for the students as they include simple questions of intuitive answer and of which learners already have experience in paper. The cultural matter is also important because Chinese culture grants a great importance to the handwriting: the visual beauty of ideograms, the technique of its accomplishment and the metaphysical rules of the Chinese traditional culture. Time that students will take to integrate the computer in the language learning process will depend, to a large extent, on the students progress and improvement in the writing skills in Spanish and also on the personal confidence gained by taking online language tests. This first approach sets some simple guidelines to (i) continue investigating in the technological and cultural elements that are specifically linked to the reason why Chinese students prefer to use paper instead of computers when dealing with the Spanish language, as well as to (ii) identify what technological elements make students feel more insecure, and finally (iii) to what extent the cultural factor determines the election of either pen-and-paper or computer in language testing. Acknowledgements The researchers would like to thank the Ministry of Education for sponsoring this research project through its 2007 Research and Development program (HUM C02-01). References [1] J. García Laborda and J. Pardo Suárez, Diseño de materiales audiovisuales en plataformas de exámenes de idiomas, en Leonor Pérez Ruiz, Isabel Pizarro Sánchez, Elena González-Cascos, Estudios de metodología de la lengua inglesa (IV), , 2008 [2] H. Breland, Y. W. Lee, and E. Muraki, Comparability of TOEFL CBT Essay Prompts: Response-Mode Analyses, Educational and Psychological Measurement, v65 n4 p Aug 2005 [3] J.García Laborda and T. Magal Royo, Diseño de materiales audiovisuales en plataformas de exámenes de idiomas, Ibérica: Revista de la Asociación Europea de Lenguas para Fines Específicos (AELFE), ISSN , Nº. 14, 2007, pags [4] E. Hansen, D. Forer, and M. Lee, A "Self-Voicing" Test for Individuals with Visual Impairments. Journal of Visual Impairment & Blindness, v96 n4 p Apr 2002 [5] C. Taylor, J. Jamieson, D. Eignor, and I. Kirsch, The name assigned to the document by the author. This field may also contain sub-titles, series names, and report numbers, Language Learning, v49 n2 p Jun 1999 [6] E. Wolfe and J. Manalo, Composition medium comparability of direct writing assessments for non- native English speakers. Language Learning and Technology, 8, Available from [7] Porter, R. y L. Samovar, An introduction to intercultural communication Wadsworth Publishing Company, Belmont, California, [8] R. Ellis, Second Language Acquisition Oxford University Press, 1997 p.4. [9] J.P. Gaudron, E. Vignoli, Assessing computer anxiety with the interaction model of anxiety: development and validation of the computer anxiety trait subscale. Computers in Human Behavior, 18, , [10] B. Smith and P. Caputi, Cognitive interference model of computer anxiety: Implications for computer-based assessment. Computers in human Behaviour, 23, , [11] D. Mahar, R. Henderson and F. Deane, The effects of computer anxiety, state anxiety, and computer experience on users' performance of computer based tasks. Personality and individual differences, 22, , [12] Instituto Cervantes, Plan Curricular del Instituto Cervantes. Niveles de referencia para el Español. Vols. 1-3, Edelsa, Madrid, 2007 [13] D. Bennett, Reinventing assessment: speculations on the future of large-scale educational testing. Princeton, NJ: Educational Testing Service, Policy Information Center, [14] A. Gimeno, New challenges in developing an online CALL authoring shell, content manager and courseware: the INGENIO model. EUROCALL Newsletter, Nº 7, May