Report for a General Audience

To Investigate Preliminary Claims of Linkage between the Graduating Students Language Proficiency Assessment (GSLPA) (English) and the Common European Framework of Reference for Languages (CEFR) Report for a General Audience

Introduction This report summarises the results of a project designed to explore linkage between the English version of the Graduating Students Language Proficiency Assessment (GSLPA) and the Common European Framework of Reference for Languages (CEFR). The report is written for a general audience, which may include test takers and test users such as universities and employers. The report will provide a general description of the GSLPA; information about the processes carried out to link the GSLPA to the CEFR; and a profile of the relationship of the GSLPA to the CEFR levels. The project was carried out at the Hong Kong Polytechnic University under the auspices of the English Language Centre between January 2008 and August 2009. A full technical report on the project can be found on the GSLPA website at www.elc.polyu.edu.hk/gslpa. The Graduating Students Language Proficiency Assessment The GSLPA consists of tests of written and spoken English, written Chinese and its spoken equivalent, Mandarin Chinese or Putonghua. The tests are designed to measure the abilities of university students to function effectively in these areas in the kinds of workplaces that they are likely to find themselves in once they graduate. The tests were developed during the 1990s at the Hong Kong Polytechnic University and first administered to students in 1999. The GSLPA (English) is administered once per year and is a compulsory requirement for all students at the University. The GSLPA Written Language Test consists of two tasks: Task A Task B A piece of writing representing a correspondence internal to a workplace, e.g. an email, a memo or a letter. Word length is approximately 150 words and content is provided. A piece of writing representing a correspondence (normally though not restricted) to a second party outside of the workplace. Word length is approximately 200 words and content requires more input from test taker. The Test is administered on paper in a large hall under examination conditions. Test papers are scanned for storage and marking. The GSLPA Spoken Language Test consists of five tasks: Task 1 Task 2 Task 3 Task 4 Task 5 A two-minute summary of a simulated radio interview between two speakers lasting five-six minutes. An interview consisting of four questions with 40 seconds to answer each question. A three-minute presentation of information provided in a table. A voicemail message of 40 seconds in response to a voicemail asking for discrete items of information. A three-minute turn in a conversation about an aspect of life in Hong Kong.

The Test is administered in a language laboratory under examination conditions. Candidates recordings are uploaded to a secure server then distributed for marking. A general description of the GSLPA is provided below. Further information on the GSLPA can be found on their website at www.elc.polyu.edu.hk/gslpa. GENERAL EXAMINATION DESCRIPTION General Information Name of examination Language tested Examining institution Versions analysed (date) Type of examination Purpose Target population No. of test takers per year Graduating Students Language Proficiency Assessment English The Hong Kong Polytechnic University 2007 International National Regional Institutional Test takers ability to use language in professional situations Lower Sec Upper Sec Uni/College Students Adult 3300 What is/are principal domain(s)? Which communicative activities are tested? Public Personal Occupational Educational 1 Listening comprehension 2 Reading comprehension 3 Spoken interaction 4 Written interaction 5 Spoken production 6 Written production 7 Integrated skills 8 Spoken mediation of text 9 Written mediation of text 10 Language usage 11 Other: (specify): Name of Subtest(s) Spoken Language Test Spoken Language Test Written Language Test Spoken Language Test Written Language Test Give the name and duration of test subtests Name of Subtest 1. Written Language Test 2. Spoken Language Test Duration of Subtest 1 hour 20 minutes 1 hour 3

GENERAL EXAMINATION DESCRIPTION What type(s) of test tasks are used? What information is published for candidates and teachers? Multiple-choice True/False Matching Ordering Gap fill sentence Sentence completion Gapped text / cloze, selected response Open gapped text / cloze Short answer to open question(s) Extended answer (text / monologue) Interaction with examiner Interaction with peers Other Overall aim Principal domain(s) Test subtests Test tasks Sample test papers Video of format of oral Subtests used in (Write numbers above) 2 1 2 Sample answer papers Marking schemes Grading schemes Standardised performance samples showing pass level Sample certificate What is reported? Global grade Grade per subtest Global grade plus graphic profile Profile per subtest The Common European Framework of Reference for Languages (CEFR) The CEFR is a Council of Europe initiative, which provides a common basis for language syllabuses, curriculum guidelines, examinations, textbooks, etc. It describes what language learners have to learn to do in order to use a language for communication and what knowledge and skills they have to develop so as to be able to act effectively. The CEFR is published in a number of languages, including English. More information on the CEFR can be found by visiting the Council of Europe website at http://www.coe.int/t/dg4/linguistic/cadre_en.asp. The CEFR defines six levels of proficiency, from C2, the highest to A1, the lowest, which allow learners progress to be measured at each stage of learning and on a life-long basis. The six levels are illustrated by the global scale shown below. 4

The CEFR Global Scale CEFR Level Proficient User Independent User Basic User C2 C1 B2 B1 A2 A1 CEFR Global Descriptor Can understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shade of meaning even in more complex situations. Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express him/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices. Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without strain for either party. Can produce clear, detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options. Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes & ambitions and briefly give reasons and explanations for opinions and plans. Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very basic personal and family information, shopping, local geography, employment). Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can describe in simple terms aspects of his/her background, immediate environment and matters in areas of immediate need. Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help. 5

Methodology The methodology employed in the project followed the procedures recommended by the Council of Europe. An overview of the procedures carried out is shown below. OVERVIEW OF THE PROCEDURES TO LINK EXAMINATIONS TO THE CEFR LEVELS: A: SPECIFICATION: Relating to the CEFR through specification of examination content Internal familiarisation with CEFR Individual With colleagues Analysis on the basis of general examination description and global CEFR scale Analysis on basis of a description of different communicative activities tested (e.g. reading, spoken interaction, etc.) and the relevant CEFR scales Analysis on the basis of a description of the specific aspects of communicative language competence tested (e.g. vocabulary range, spoken fluency, etc.) and the relevant CEFR scales B: STANDARDISATION Relating to the CEFR through training with standardized tasks and samples of performance Familiarisation training Training in assessing performances in relation to the CEFR levels using standardized samples Benchmarking local samples to the CEFR levels Training in judging the CEFR level of test items using standardised samples Benchmarking local tests to the CEFR levels System of regular standardisation meetings C: EMPIRICAL VALIDATION Relating to the CEFR through empirical validation of the standards arrived at though A and B. Quantitative analysis of test quality Qualitative analysis of test quality Validation to a test acting as external criterion Validation to a set of judgements in a performance assessment acting as external criterion Validation to a set of teacher judgments acting as external criterion 6

Outcomes The outcomes of the project, i.e. the proposed linking between the GSLPA and the CEFR for spoken and written language are shown below. Profile of the Relationship of the GSLPA to CEFR Levels CEFR Levels GSLPA Spoken Language Levels GSLPA Written Language Levels C2 C1 5+ to 6 5 or above B2+ 4+ to 5 4+ to 5 B2 4 to 4+ 4 to 4+ B1+ 3+ to 4 3+ to 4 B1 3 to 3+ 3 to 3+ A2+ 2+ or below 2+ or below A2 A1 For both spoken and written English, the GSLPA was determined to be situated between CEFR Levels B1 and C1. Differences in the nature of the tasks that candidates are asked to complete in the two GSLPA tests account for the differences in the relationships with the CEFR for spoken and written English. Conclusions Preliminary linkage between the GSLPA and the CEFR has been completed. Inherent differences between the GSLPA, which is a workplace-oriented assessment designed for university graduates in the Hong Kong context, and the CEFR, which is a framework for language learning, teaching and assessment in the European context, mean that a relationship between them can only be described as indicative. However, the process of exploring the relationship has shown that the GSLPA is a valid and reliable assessment which can be considered to have international transferability. The relationship will continue to be further developed and strengthened; enabling test users to better understand what the results of the GSLPA mean and how they can best be used. Language Testing Unit English Language Centre Hong Kong Polytechnic University November 2009 7