THE USE OF INTERNATIONAL EXAMINATIONS IN THE ENGLISH LANGUAGE PROGRAM AT UQROO

THE USE OF INTERNATIONAL EXAMINATIONS IN THE ENGLISH LANGUAGE PROGRAM AT UQROO M.C. José Luis Borges Ucan M.C. Deon Victoria Heffington Dr. Alfredo Marín Marín Dr. Caridad Macola Rojo Universidad de Quintana Roo Abstract According to Cambridge ESOL in order to reach the B1 level of the Common European Framework of Reference students may need approximately 350 400 hours of guided study. In the first phase of this work, the main aim was to find out whether students in the English language program would pass the PET exam after 384 hours of guided study and identify their strengths and weaknesses in language skills. In a second phase of the study, the TOEFL examination was applied to the same group of students. The results in both examinations were carefully analyzed in order to define the level of the development of the skills. 1. INTRODUCTION Evaluation plays an important role in student s achievement while learning a foreign language. Furthermore, when it comes to comparing examinations, it is not easy since most of them have different emphases (Simmonds, 1985). Despite this fact, the need for examining student s performance in different language skills still remains. It is assumed that a significant number of students enrolled in the English Language Major at Universidad de Quintana Roo do not develop their language skills at the same level of competence, and therefore, are more likely to fail at achieving a high score on an international examination, such as Preliminary English Test (PET) or Test of English as a Foreign Language (TOEFL) after receiving 384 hours of guided study. The main aim of this study is not only to describe students English level according to different international sample tests, but identify in which skills they 648

have more strengths and weaknesses. Additionally, a comparison of students scores among two international sample tests will be presented. 2. BACKGROUND 2.1 Common European Framework of Reference for Languages (CEFR) The Common European Framework of Reference for Languages (CEFR) provides a common basis for the elaboration of language syllabuses, curriculum guidelines, examinations, textbooks, etc. It describes in a comprehensive way what language learners have to learn to do in order to use a language for communication and what knowledge and skills they have to develop so as to be able to act effectively. The Framework also defines levels of proficiency which allow learners progress to be measured at each stage of learning and on a life-long basis. It provides the means for educational administrators, course designers, teachers, teacher trainers, examining bodies, etc., to reflect on their current practice, with a view to situating and co-ordinating their efforts and to ensuring that they meet the real needs of the learners for whom they are responsible. 2.1.1 Cambride ESOL PET examination Cambridge ESOL Exams are aligned to the CEFR, which makes it easier for both learners and teachers to see the level of different qualifications, identify which exam corresponds to each level, and measure the progress they are achieving. According to Figure 1.1, a Threshold user (B1) should be able to pass the Preliminary English Test (PET). 649

Figure 1.1 Cambridge ESOL exams aligned to CEFR According to Cambridge ESOL, students who are at level B1 have been exposed to approximately 350-400 hours of guided study and are capable of performing the following tasks: ALTE Can Do Statements: overall general ability CEFR LEVEL B1 Listening/Speaking Reading Writing CAN express opinions on CAN understand CAN write abstract/cultural matters in a routine information letters or make limited way or offer advice and articles, and the notes on within a known area, and general meaning of familiar or understand instructions or non-routine predictable public announcements. information within a matters. familiar area. Figure 1.2 ALTE Can Do Statements for B1 (Threshold) level 650

2.1.2 ELT Internet-based Test of English as a Foreign Language (TOEFL ibt ) The Internet-based TOEFL ibt (Test of English as a Foreign Language) measures the ability to use and combine listening, reading, speaking and writing skills to communicate in an academic environment. ELT offers the following minimum scores on the TOEFL ibt corresponding to each level of the CEFR. Minimum scores were identified separately for speaking, listening, writing and reading: Test section Total Score Range Scale Minimum score B1 Level TOEFL ibt total 0-120 57-86 TOEFL ibt Reading 0-30 8 TOEFL ibt Listening 0-30 13 TOEFL ibt Speaking 0-30 19 TOEFL ibt Writing 0-30 17 Source: ETS TOEFL Likewise to Bachman, et al (1995) this is a comparative study between two sample international tests which are used worldwide to assess students general English. 3. METHODOLOGY 3.1 Participants In phase one of the study we decided to apply the PET (Preliminary English Test) to 33 students who had finished the English III course in the program. These students had already had 384 hours of guided instruction, and therefore, were supposed to reach the PET, i.e. level B1 according to the Common European Framework for Languages. The participants were all Mexican and Spanish native speakers and also enrolled in the English Language Program at the University of Quintana Roo. 651

3.2 Instruments Two instruments were used in this study: the Preliminary English Test from Cambridge ESOL examinations and the TOEFL ibt from Educational Testing Services (ETS). A comparison of the results in both tests will be provided in the following paragraphs. 3.2.1 PET The PET test used can be found at the official web page of Cambridge ESOL examinations. They are sample tests provided by Cambridge and can be considered as valid and reliable as an official PET examination. The test contains four sections or papers: reading, writing, listening and speaking. They all have the same weight which is 25%. 3.2.2 TOEFL The Test of English as a Foreign Language (TOEFL) is used worldwide to measure English Language Proficiency. We used the companion CD-ROM of the Official Guide to the New TOEFL ibt which includes authentic sections. The TOEFL examination also gives the same weight to all four skills. The score scale is 0-30 for each skill making a total score of 0-120. 3.3 DATA COLLECTION 3.3.1 Applying the PET The examiners had to become familiar with the test format of the Preliminary English Test (PET) especially for the evaluation of speaking and writing in which both examiners were asked to provide a score. One of the main problems encountered during the application of the PET was that the participants were hard to locate and finally only 22 out of 33 completed the four sections of the test. Therefore, only data obtained from 22 was analyzed. 652

3.3.2 Applying the TOEFL We decided to used the TOEFL ibt because it measures the same four abilities: reading, listening, speaking, and writing. Other versions of the TOEFL examination omit the evaluation of writing and/or speaking skills. The main problem encountered during the application of the TOEFL was that some students had busy schedules which made it difficult for them to take an examination that lasted approximately four hours. In the end, only 11 students out of 22 were able to complete the reading and listening sections. 1. RESULTS AND DISCUSSION 4.1 PET scores On a PET exam, a student may achieve any of the following scores: Pass with Merit =>85% Pass 70-84% Narrow Fail 65-69% Fail =<64 After grading the 22 examinations we obtained the results explained in the following table. Frequency Percent Pass with merit 1 4.5 Pass 12 54.5 Narrow fail 3 13.6 Fail 6 27.3 Total 22 100.0 Figure 1.3 Results from the PET As it can be observed in figure 1.3, 59.1% of the total number of participants passed the examination and 40.9% failed. 653

4.2 TOEFL scores As mentioned above both tests use different scales, and in the case of TOEFL it was necessary to have a reference that would allow us to find out whether a student reached or did not reach the B1 level of the Common European Framework for Languages. In a study conducted by ETS researchers Richard J. Tannenbaum, Ph.D., and E. Caroline Wylie, Ph.D., in October 2006, a panel of 23 experts was able to make connections between the TOEFL and the CEFR. The table below presents the recommended score ranges and the minimum score for each modality at each CEFR proficiency level according to that study. By using the table above we found out the following results for each participant and we compared them with the overall result from the PET. ibt ibt Subject Result PET Reading Result Listening Result 1 B1 16-18 B1 21-23 B2 2 Fail 15-18 B1 10-15 B1 3 B1 18-21 B1 16-21 B1 4 B1 15-18 B1 16-18 B1 5 Fail 14-16 B1 9-13 B1 6 B1 22-24 B2 17-22 B1 7 B1 25-27 B2 26-29 C1 8 B1 16-18 B1 14-18 B1 9 B1 19-21 B1 6-12 Fail 10 B1 12-16 B1 14-18 B1 11 B1 15-18 B1 10-15 B1 Figure 1.4 Results from the PET compared with results from the TOEFL 654

Results show that some participants did not reach B1 level according to the PET but they did reach it according to the TOEFL in reading and listening. Case number one, six and eight reached a higher level in this examination. Case nine failed to pass the listening section despite having reached level B1 in the first examination. 5. CONCLUSIONS AND IMPLICATIONS In the first phase of the study we expected students to perform better in receptive skills. However, the results show that students achieved a lower skill in listening than in any other skill. The reasons may be many but we assumed that the fact that students are not exposed to the British accent during the first three courses could lead to poor comprehension in the listening section. Therefore, it is suggested that students be exposed to different varieties of English throughout their courses of English. Another reason may be that listening requires more development of short-term memory than other skills. The results show that skills are not developed at the same level. This is probably due to the lack of equal distribution of time to reinforce each skill in the classroom. Further research about time used to develop different skills can be made by using class observations. It is recommended that students be given an individual and overall grade. This will help both the instructors and students find a more balanced development of skills. Students performed significantly better in the speaking section (18.6%). This result shows a possitive washback effect, in terms of the communicative approach that is used in English classes at UQROO. In the second phase of the study, we found that students tend to perform better in the TOEFL than in the PET. Further research can be done in order to find out a more accurate table of equivalences than the one currently used in the department of language and educational. A limitation of this study was the number of participants. A much larger number will be necessary in order to improve the current table of equivalences. 655

Finally, it is important to notice that the students level of English needs to be improved in the first semesters and that the proficiency of the students needs to be measured not only by using achievement tests but also proficiency tests. Using the Cambridge or the ETS examinations after certain semesters may be an excellent tool to measure the progress of the students and might even help them prepare to reach the C1 level, which is a requirement to obtain their degree. 656

References 1. Bachman, L. et al (1995). An Investigation into the Comparability of Two Tests of English as a Foreign Language: The Cambridge-TOEFL Comparability Study. Cambridge: Cambridge University Press 2. CEFR and Can Do. Retrieved October 5, 2009, from http://www.cambridgeesol.org/exams/exams-info/cefr.html 3. Common European Framework of Reference for Languages. Retrieved October 3, 2009, from http://www.coe.int/t/dg4/linguistic/source/framework_en.pdf 4. Mapping TOEFL ibt on the Common European Framework of Reference. Retrieved October 3, 2009, from http://www.ets.org/media/research/pdf/cef_mapping_study_interim_report.pdf 5. Simmonds, P. (1985). A Survey of English Language Examinations. ELT Journal, 39 (1), 33-42. Retrieved October 1, 2009, from ELT Journal on CD-ROM 6. The TOEFL Test - Test of English as a Foreign Language. Retrieved October 3, 2009, from http://www.ets.org/portal/site/ets/menuitem.fab2360b1645a1de9b3a0779f1751509/? vgnextoid=69c0197a484f4010vgnvcm10000022f95190rcrd 657

Biodata José Luis Borges Ucán is a teacher and researcher at Universidad de Quintana Roo. He holds a BA in English Language from Universidad de Quintana Roo and an MA in Curriculum and Instruction from University of Nebraska. Contact: jlborges@uqroo.mx Deon Victoria Heffington is a teacher and researcher at Universidad de Quintana Roo. She holds a BA in English Language and MA in Education. Her research interests include individual differences in language learning. She is currently head of the Department of Language and Education. Contact: deon@uqroo.mx Alfredo Marín Marín is a teacher and researcher at the Universidad de Quintana Roo. He holds a BA in English Language and a especialidad from the Universidad Veracruzana, an MA in Education from the Universidad de La Habana, and a Ph.D. in English Language Teaching from the University of Essex, UK. His research interests include individual differences in language learning, teachers/learners beliefs, L2 vocabulary acquisition, and uses of corpora in ELT. Contact: alfmarin@uqroo.mx Caridad Macola Rojo holds a BA in the Teaching of English from the Havana Business University and Doctor in Education from the University of Havana. Fulltime professor in the Language and Education Department in the University of Quintana Roo. Has written in co-authorship textbooks for the teaching of English, Translation and English Grammar. Is recently involved in research concerning the development of skills in language learning, especially the skill of reading. Contact: macola@uqroo.mx 658