X Seminario de Administradores de Programas Educativos de Cooperación n Internacional

Transcription

1 X Seminario de Administradores de Programas Educativos de Cooperación n Internacional La evaluación n de las lenguas extranjeras: modelos para la certificación n del dominio del nivel del idioma de mayo de 2009 Dpto. de Certificación Lingüística Standards in Spanish language testing: a global perspective Standardisation and globalisation The development of standards and scales for language assessment The Common European Framework of Reference International quality standards Language certification at the Instituto Cervantes Instituto Cervantes 1

2 Standardisation and globalisation Artificial (but rational) vs. Natural (solar time). Synchronising communities. Uniform standard, flexibly applied and differentially experienced. Globalisation: quest for international standards (transparency, accessibility, communication, empowerment of stakeholders). Tension: linguistic / cultural homogeneization and protection of diversity. The development of standards and scales for language assessment Background Treasure Didactica magna ( ) Palace Gate Vestibule Comenius European scales Foreign Service Institute Interagency Language Roundtable Instituto Cervantes 2

3 The development of standards and scales for language assessment Ambilingual Proficiency Comprehensive Operational Proficiency European scales Wilkins 1978 Adequate Operational Proficiency Limited Operational Proficiency Basic Operational Proficiency (Threshold) Survival Proficiency Formulaic Proficiency British National Language Standards Eurocentres Scale of Language Proficiency ( ) The Swiss Project ( ) (1997- ): Socrates Program LINGUA Can-do Project (1994- ) Common European Framework of Reference The development of standards and scales for language assessment C2 C1 B2 B1 A2 A1 Mastery Effective Operational Proficiency Vantage Threshold Waystage Breakthrough Superior Advanced High Advanced Mid Advanced Low Intermediate High Intermediate Mid Intermediate Low Novice High Novice Mid Novice Low ILR Level 5 ILR Level 4 ILR Level 3 ILR Level 2 ILR Level 1 Native or bilingual proficiency Full professional proficiency Professional working proficiency Limited working proficiency Elementary proficiency 12 Benchmarks in 4 skills Instituto Cervantes 3

4 The development of standards and scales for language assessment International Civil Aviation Organization ICAO Level 6 ICAO Level 5 ICAO Level 4 Expert Extended Operational Canadian Academic English Language Assessment Expert User Adept User Very Competent User Competent User Competent but Limited User Marginally Competent User Limited User Very Limited User Non User STANAG 6001 Level 5 STANAG 6001 Level 4 STANAG 6001 Level 3 STANAG 6001 Level 2 STANAG 6001 Level 1 STANAG 6001 Level 0 ICAO Level 3 ICAO Level 2 ICAO Level 1 STANAG 6001 Pre-Operational Elementary Pre-Elementary Highly articulate native Expert Professional Functional Survival No proficiency The development of standards and scales for language assessment Popularity of scales and their descriptors: Stereotypes that language users can compare themselves with Help to increase reliability of non-objective scores Guidelines for the design and development of tests Help to report and interpret results Provide internal consistency to academic activities in educational establishments Enable comparison of educational systems or student populations Serve as benchmarks for high-stakes decision-taking Instituto Cervantes 4

5 Based on a model of communicative competence Descriptors have been carefully worded Based on Rasch measuring model Methodology based on empirical field work Intuitive Phase: Creating a pool of classified, edited descriptors Qualitative Phase: Analysis of teachers discussing proficiency 32 teacher workshops sorting descriptors Quantitative Phase: Teacher assessment of 2,800 learners on descriptor checklists (300 teachers) Teacher assessment of videos of some learners Interpretation Phase: Setting cut-points for common reference levels A1 A2 B1 B2 C1 C2 Common European Framework of Reference Breakthrough Acceso Waystage Plataforma Threshold Umbral Vantage Avanzado Effective Operational Proficiency Dominio operativo eficaz Mastery Maestría Instituto Cervantes 5

6 Overview design programs Course designers transparency develop materials Materials writers learn autonomously encourage autonomous learning coherence design curricular systems Teachers Education develop authorities certification systems develop exams Examiners Students Assessment bodies C OMM O N B ASE Scales before CEFR: Wording tended to be relative or elusive. The descriptors were seldom stand-alone criteria (i.e. discrete). Wording often created semantic appearance of a scale, without actually describing anything. The situation of descriptors at a particular level was arbitrary, following convention / cliché. Lower levels tended to be worded negatively. CEFR Levels: Key Problems > Danger of differing interpretations for different languages. > Under-definition of C2; some reversals of C1/C2 descriptors (ALTE, DIALANG). > Weak definition of socio-linguistic competence. > Unrealistic expectations in relation to receptive skills. Learning, Teaching, Assessment Instituto Cervantes 6

7 What the CEFR is not A harmonisation tool We have NOT set out to tell practitioners what to do or how to do it. We are raising questions not answering them. A theory of language or skills development Scales describe learning outcomes, learner behaviours, not the invisible processes involved. A test specification Scales and lists can be consulted when drawing up a task specification (Ch. 4) or defining assessment criteria (Ch. 5) but need reference to detailed specifications for language and context. Life beyond C2 D1 D2 E Common European Framework of Reference Language professionals: Interpreters, translators, some university professors Ambilingual Genuine bilinguals WENS Well-educated native speaker Instituto Cervantes 7

8 Manual for Linking Examinations to the CEFR (January 2009) BASIS FOR CLAIM Recommended procedures encourage alignment of examinations to CEFR at differing degrees of rigour appropriate to different testing contexts and to the extent permitted by resources and expertise available. THEORETICAL FRAMEWORK SPECIFICATION Familiarisation with CEFR Internal validity. Description and analysis of test quality: Test contents Test development process Grading and results Test Test analysis and review STANDARD SETTING Standardisation of judgments Training in assessing performances in relation to CEFR levels (using illustrative samples) Training in judging the difficulty of test items in relation to CEFR illustrative items VALIDATION Data collection Test validity: Content validity Operational aspects Psychometric aspects Description of the test in relation to the CEFR Overall estimation of examination level Communicative activities tested Aspects of communicative language competence tested Graphic profile of relationship of the test to CEFR Benchmarking local performance samples to CEFR levels Assigning local items and tasks to CEFR levels Establishing cut-off scores Standard setting validity: Procedural validity Internal validity External validity Documented CLAIM of link to CEFR based on specification Documented CLAIM of link to CEFR based on specification and standardisation Documented CLAIM of link to CEFR based on empirical verification International quality standards Codes of practice ILTA: International Language Testing Association Test development Test administration Rater training Quality management systems Attaining and maintaining quality Striving for fairness Test review and analysis Linkage to CEFR Ethics: washback and stakeholders Instituto Cervantes 8

9 Language certification at the Instituto Cervantes Marco Común Europeo (Consejo de Europa) ) y Marco ALTE A1 Breakthrough (Acceso) A2 Waystage (Plataforma) B1 Threshold (Umbral) B2 Vantage (Avanzado) C1 Effective Operational Proficiency (Dominio operativo eficaz) C2 Mastery (Maestría) a) (Nivel A1) (Nivel A2) (Nivel B1) (Nivel B2) (Nivel C1) (Nivel C2) Diplomas de como Lengua Extranjera Plan Curricular del Instituto Cervantes Cursos de nivel A1 Cursos de nivel A2 Cursos de nivel B1 Cursos de nivel B2 Cursos de nivel C1 Cursos monográficos y de extensión Language certification at the Instituto Cervantes Localisation of assessment services: DELE-F EPSO ESLC CLIC CIE French Ministry of Education contract: : Special DELE examinations: : CEFR linked A2-B1; twin certification Students from European Lyceums: : candidates annually Test development and rater training European Personnel Selection Office (EPSO) Spanish in professional contexts; ; CEFR linked A2 / B2; promotion in the workplace Test development and rater training at IC Brussels European Survey on Language Competence: : CEFR linked SurveyLang consortium: : Consulting Certificados de Competencia Lingüí üística IC: for visa applications only CEFR linked: levels A2 to B2 (oral skills) Test development and rater training Certificado Internacional de : IC UNAM Spanish in academic contexts; ; CEFR linked: levels A2 to C1 (4 skills); computer-based based; diagnostic; university entry requirement Test administration tool and rater training Instituto Cervantes 9

10 Language certification at the Instituto Cervantes SICELE Sistema Internacional para la Certificación del como Lengua Extranjera Harmonisation of criteria for Spanish language certification. International standards: development, administration of tests and issue of certificates. Verification of standards. Reciprocity in the official recognition of Spanish language certificates throughout the Spanishspeaking world. Respect for diversity: language varieties in Spanish. Unidad en la diversidad. 4 national governments: Argentina, Colombia, Mexico and Spain. 60 universities from Spain and Spanish America. Conferencia de Rectores de las Universidades as (CRUE). Asociación Colombiana de Universidades (ASCUN) Consejo Nacional de Educación Superior del Ecuador. Consejo de Rectores de Panamá. Asociación Venezolana de Universidades. Back to standardisation and globalisation Conventions adopted internationally are useful to citizens worldwide... BUT solutions have to be provided locally. International assessment bodies can offer benchmarked testing products... BUT they must be prepared to offer specific solutions to local demands. A uniform standard, flexibly applied and differentially experienced. Instituto Cervantes 10