COMPUTER-BASED TESTING IN A COURSE OF ENGLISH FOR SPECIFIC PURPOSES Daniela Dvořáková Language Centre at the Faculty of Education Masaryk University
CONTENT EAP, ESP in tertiary education EAP, ESP courses at the Faculty of Education Projects COMPACT, IMPACT Stages of test construction Marking Pretesting and analysis of test results Questionnaires Benefits and possible challenges of computerbased tests Conclusion
ESP, EAP IN TERTIARY EDUCATION Roles of educational institutions: English as a Foreign Language (EFL) VERSUS English for Academic Purposes (EAP), English for Specific Purposes (ESP) Language for Specific Purposes (LSP) J.G. Laborda (2001): the teaching of a language as a second or foreign language for certain groups of students to whom the syllabus, tasks and methodology is especially tailored to their interests and needs. The main aim of EAP Laurillard (2002): Prepare students for acquiring, internalizing and extending subject-knowledge during lectures, seminars, discussions and while reading/producing specialist texts in English by: teaching academic language and developing academic skills; developing students attitudes towards learning and teaching by expanding their metacognitive strategies; promoting students critical thinking and ability of expressing their opinions on specialist topics.
LANGUAGE INSTRUCTION AT THE FACULTY OF EDUCATION MU 51 non-philological Bachelor s study programmes 57 non-philological Master s study programmes courses in English, German non-homogenous groups of students Bachelor s programmes two semesters of compulsorily-optional courses A and B (examination at B1 level of the CEFR after the 2nd semester of studies); length of one seminar 90 minutes a week. Master s programmes two semesters of compulsorily-optional courses C and D (examination at B2 level of the CEFR after the 2nd semester); length of one seminar - 90 minutes a week.
PROJECTS COMPACT AND IMPACT Project COMPACT (Competences in Language Education at Masaryk University) financed by the EU, the Ministry of Education, the CR (Operational Programme Education for Competitiveness) May 2009 May 2012 AIM: to standardize and innovate language courses focused on developing both general academic and specific academic skills and abilities of MU students. Project IMPACT (Innovation, methodology and the quality of language teaching and specialist education in foreign language teaching in tertiary education in the Czech Republic) June 2012 June 2015 AIM: to support language teaching methodically at universities; to innovate some specific courses; to standardize the testing and oral examining of Masaryk University students; to assess language competencies in general and specific academic languages; to educate further university language teachers and language methodologists.
STAGES OF TEST CONSTRUCTION Test specifications Item writing Moderation
TEST SPECIFICATIONS The test purpose The test is used at the end of the second term of courses of English for Teachers and English for Special Teachers. It is designed to assess required language competences at level B2 of the CEFR and to give students a credit. Characteristics of tested language and test level language for specific and academic purposes at level B2 of the CEFR, specialist terminology for individual disciplines, academic skills at level B2, written and oral language. Test type proficiency 55%: listening, reading, writing, 50% of speaking; achievement 45 %: use of English; computer-based. Topics o related to education, teaching and professional career, special education, social education, didactics, psychology. Pass marks o o standard setting; no pass marks for each subtest. Criteria and methods of correcting o o computer-based correcting; teachers.
LANGUAGE SKILLS TO BE TESTED Language skills Marks Length Terminology 10 % 8 min. Listening 14 % 16 min. Reading 12 % 24 min. Use of English 20 % 15 min. Writing 14 % 20 min. Speaking 30 % 15 min.
SUMMARY OF WHAT IS IN THE EXAM Task Type of task Skills assessed 1 Listening MCQ Listening for the general meaning of the whole text. 2 Listening gap-fill Listening for information, opinion or detail. 3 Reading multiple matching Reading for understanding the main idea of a text. 4 Reading - MCQ Reading for detailed comprehension, understanding opinions and feelings, understanding how a text is organised. 5 Use of English MCQ Tests knowledge of grammar at level B2. 6 Use of English sentence transformations 7 Use of English word formation 8 Translation from Czech into English Tests vocabulary and grammar when paraphrasing. Tests knowledge of vocabulary and grammar when forming different parts of speech. Test knowledge of special terminology. 9 Writing Tests different aspects of writing a discursive composition, such as grammar, vocabulary at level B2, how well students organize their ideas.
MARKING Two types: OBJECTIVE and SUBJECTIVE Objective marking terminology, listening, reading, MCQ grammar, word formation, sentence transformations. mechanical, carried out by a computer. KEY and MARK SCHEME Key one and only one correct answer for each item: MCQ, matching Mark scheme more than one possible response for an item: sentence transformations; acceptable responses are added to the mark scheme after pretesting. Subjective marking writing and speaking; marked by a teacher; an analytic scale.
PRETESTING AND ANALYSIS OF TEST RESULTS Reasons for pretesting need to know how the test will work. Pretesting took place last spring. Test analysis the facility value, the discrimination index, the internal reliability. Examples of items with a negative D. I.: We really need her approval, so I hope she won t PUT US DOWN. (turn us down, ring us up, pick us up) Some teachers disapprove of ALTERNATING alternative methods (alternate). Results of pretesting and statistical analysis.
SURVEY OF STUDENTS: QUESTIONNAIRES Question Yes No 1 Were the instructions for all items clear? 64 0 2 Were the texts in the listening tasks interesting? 51 11 3 While listening did you have enough time to write down your answers? 4 In your opinion, were there any items where more than one of the alternatives is correct? 43 21 23 40 5 Were the texts in the reading tasks interesting? 55 10 6 Were there any items which can be answered without any reference to the reading or listening passage? 7 Comments. 15 49
STUDENTS COMMENTS Some students would prefer paper-based testing; would like to have more time for the test; would like to have a possibility to listen to the recording more than twice; consider especially listening tasks difficult. Other students appreciate a possibility to know the test results immediately after finishing the test; think the tasks and level were appropriate; think the test was well-arranged and clear.
BENEFITS AND CHALLENGES OF COMPUTER-BASED TESTING BENEFITS cost savings on printing; some improvements in test security; data are more accurately collected, easier to store; responses are scored with by the computer it can simplify the process of scoring the exam; offers immediate test scoring/immediate results; an opportunity to use innovative item types; may be preferred by some students, especially by disabled ones. CHALLENGES necessity of a sufficient number of computers; test security concerns; specific software is needed; interruption of a test by technology issues; some students may have computer anxiety.
CONCLUSION Standardized assessment of language competences in specialised areas of a foreign language Computer-based testing a modern and efficient way of testing Benefits of computer-based testing prevail over possible drawbacks
REFERENCES ALDERSON, J. C. & CLAPHAM, C. & WALL, D. 1995.Language Test Construction and Evaluation. Cambridge University Press, 1995. ISBN 0521 47255 5 DUDLEY-EVANS, T. & ST JOHN, M., J. 1998. Developments in English for Specific Purposes. Cambridge University Press, 1998. ISBN 0 521 59675 0 DUTTLINGER, C. & FANIZZA-SCHEIPER, F. & LINARES I ZAPATER, G. 2007. Manual for Students. Bratislava: Ekonóm, 2007. ISBN 978-80-225-2515-2 GREEN, R. 2013. Statistical Analysis for Language Testers. Palgrave Macmillan, 2013. ISBN 978 1 137 01828 1 HUGHES, A. 1989. Testing for Language Teachers. Cambridge University Press, 1989. ISBN 0 521 27260 2 HUTCHINSON, T. & WATERS, A. 1987. English for Specific Purposes. Cambridge University Press, 1987. ISBN 0 521 318378 HYLAND, K. 2006. English for academic purposes: An advanced resource book. London: Routledge, 2006. ISBN-13: 978 0415358705 JORDAN, R. R. 1997. English for Academic Purposes: A Guide and Resource Book for Teachers. Cambridge University Press, 1997. ISBN 0 521 55618 X LAURILLARD, D. 2002. Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies. 2nd ed. London: RoutledgeFalmer, 2002. ISBN 0 415 25679 8 THURLOW, M., LAZARUS, S. S., ALBUS, D., & HODGSON, J. (2010). Computerbased testing: Practices and considerations (Synthesis Report 78). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
THANK YOU FOR YOUR ATTENTION. ARE THERE ANY QUESTIONS?