Michael Milanovic Nick Saville Angeliki Salamoura Fiona Barker

Transcription

1 Michael Milanovic Nick Saville Angeliki Salamoura Fiona Barker

2 Cambridge English Overview Introduction to Cambridge English Developing learner corpora The English Profile Programme Automated assessment of writing and speech

3 About Cambridge ESOL Part of Cambridge Assessment ESOL s mission is: To be the experts in language assessment: delivering excellence and innovation. Started developing a level system in 1913 with Certificate of Proficiency (CPE) 12 ½ hours: phonetics, translation(to and from), essay, reading aloud, conversation, dictation, literature. 3 candidates - none of whom passed! An exam for teachers reflecting current approaches to the learning and teaching of English.

4 About Cambridge ESOL Jack Roach retired 1921 double first French and German, UCLES 1925, took over CPE in 1929 with 30 candidates. By 1939 there were 752 candidates. The level question was straight forward! The issue of multiple levels first arose in the 1930 s and Lower Certificate (LCE) was introduced in (The British Council banned it from the UK). The first version of the Preliminary English Test was introduced in 1943 to cater for large numbers of foreign servicemen needing English. Emerging concept of natural levels for organising learning, teaching and assessment (Dr Peter Hargreaves; see also Trim 2011)

5 About Cambridge ESOL 1947 CPE 2028, LCE Total CPE 6002 LCE Total 20,309 We see a growth in the use of English and by ,061 candidates and still two levels. At the same time the Council of Europe was addressing the issue of levels with the specification of the Threshold Level (Van Ek, 1976) Wilkins 8 Levels. Threshold heralded the beginning of the communicative language teaching movement and was adapted into about 30 languages.

6 About Cambridge ESOL Cambridge ESOL co-funded Threshold and Waystage 1990 (Van Ek and Trim, 1990) John Trim was Head of Linguistics at Cambridge, then Head of CILT and Director of the Modern Languages Programme of the Council of Europe for nearly 30 years. Trim wrote Vantage and Breakthrough levels funded by Cambridge and ALTE. The need for English grew globally 170,384 candidates in Level system grew to reflect this PET (1984), CAE (1991), KET (1994)

7 About Cambridge ESOL 1991 Council of Europe launches the CEFR movement and the 6 level CEFR is published in Cambridge has a 5 level system but aligns to the CEFR levels in But the challenge remains what do the levels really mean? what does aligning to them mean and how many times do you have to do it for an exam? Enter English Profile Programme - EPP

8 Cambridge ESOL today Global reputation for quality and positive impact. Offers a wide range of exams and tests for learners and teachers at many levels and for many needs (e.g. IELTS) Provides access to international standards worldwide (CEFR and ALTE) Assesses nearly four million candidates a year in 130 countries. Has 320 people in Cambridge, 25 offices overseas, operates through 3,000 examination centres, services at least 50,000 schools, has over 20,000 trained examiners. Works closely with CUP and other University departments.

9 Cambridge ESOL: what we do We have a major focus on quality with over 40 staff in our Research and Validation Department. We are the European assessment agency for English language skills. We deliver exams, benchmark systems, evaluate programmes, carry out funded research e.g. SurveyLang , Asset Languages We research the educational and social impact of our English language assessment around the world.

10 Cambridge English The concept of Cambridge English has developed over the last two or three years and we want it to be the dominant global brand. It brings together the publishing, assessment and research expertise of ESOL and CUP. Provides and umbrella for joint initiatives, CLC, EPP, exams publishing, course development. Research informed. Invites interdisciplinary collaboration within and beyond the University e.g. Department of Theoretical and Applied Linguistics, Computer Labs, Engineering Department, Language Centre...

11 Cambridge English research Developing Learner Corpora & the English Profile Programme How Cambridge researchers having been analysing learner data, including corpora, to understand progression across levels of proficiency The development of automated systems for the assessment of speech and writing based on computational techniques

12 Learner Data from Cambridge ESOL exams Flyers Movers C2 Starters Beg-A2 Breakthrough Level KET A Council of Europe Waystage level PET B Council of Europe Threshold level FCE B Council of Europe Vantage level CAE C Effective Operational Proficiency level CPE C Mastery level Increasing proficiency levels

13 Cambridge ESOL exams and the CEFR Developing Proficiency A learning ladder

14 Developing learner corpora Cambridge Learner Corpus (CLC) - largest learner corpus of English exam scripts in existence Cambridge University Press and Cambridge ESOL have been developing corpora together for more than two decades Collaboration on other corpora: Cambridge English Profile Corpus Cambridge Corpus of Academic English

15 Cambridge Learner Corpus writing scripts and related tasks from Cambridge English exams from 1993 nearly 50 million words contains nearly 500,000 pieces of writing from Cambridge ESOL exams over 130 first languages aligned to all CEFR levels (A1-C2) error-coded with over 70 error types parsed and tagged for syntactic relations (Briscoe et al.)

16 Cambridge Learner Corpus writing scripts and related tasks from Cambridge English exams from 1993 nearly 50 million words contains nearly 500,000 pieces of writing from Cambridge ESOL exams over 130 first languages aligned to all CEFR levels (A1-C2) error-coded with over 70 error types parsed and tagged for syntactic relations (Briscoe et al.)

17 CEFR The Common European Framework of Reference For Languages: Learning, teaching, assessment The Framework defines levels of proficiency which allow learners progress to be measured at each stage of learning and on a life-long basis. Council of Europe (CUP, 2001:1, our emphasis)

18 The 6 CEFR levels Proficient User Independent User Basic User C2 Mastery C1 Effective Operational Proficiency B2 Vantage B1 Threshold A2 Waystage A1 Breakthrough

19 CEFR A common scale of language ability levels A1 C2 Beginner to mastery Deliberately underspecified applies to all languages A can do approach

20 CEFR Can-do statements for Writing A2 can write short, simple notes and messages relating to matters of everyday life can give short, basic descriptions of events and activities B1 can write personal letters describing experiences, feelings and events in detail can describe dreams, hopes and ambitions B2 can synthesize information and arguments from a number of sources can construct a chain of reasoned argument

21 What do these general and subjective words mean for English? Which grammatical, lexical and other properties of English does the learner know and use at each level to realise these Candos? Which more specific language functions does the learner perform with these linguistic properties at each level and how well?

22 What do these general and subjective words mean for English? Reference Level Descriptions (RLDs) for national and regional languages address this question The process of developing RLDs is referred to as transposing the CEFR to make it language specific

23 English Profile (EP) Collaborative programme, interdisciplinary research Main EP question What is learner English like at each of the 6 CEFR levels? Goal To provide Reference Level Descriptions (RLDs) for English for all 6 CEFR levels RLD = detailed information about the language learners are expected to have acquired at each level

24 The English Profile founder members University of Cambridge Cambridge ESOL Cambridge University Press (CUP) Department of theoretical and applied linguistics Prof John Hawkins, Dr Henriette Hendriks and colleagues University of Cambridge Computer Laboratory Prof Ted Briscoe and colleagues University of Bedfordshire (CRELLA) Prof Cyril Weir, Dr Tony Green and colleagues British Council English UK Council of Europe - Dr John Trim - observer

25 The English Profile founder members University of Cambridge Cambridge ESOL Cambridge University Press (CUP) Department of theoretical and applied linguistics Prof John Hawkins, Dr Henriette Hendriks and colleagues University of Cambridge Computer Laboratory Prof Ted Briscoe and colleagues University of Bedfordshire (CRELLA) Prof Cyril Weir, Dr Tony Green and colleagues British Council English UK Council of Europe - Dr John Trim - observer

26

27

28

29 EP Research Programme How do the different kinds of criterial features (lexical semantic, syntactic, discourse etc.) cluster together to define learner profiles in English? Which linguistic features realise which language functions across the CEFR levels? How does the profile of the learner vary depending on their L1? What are the pedagogical implications of such L1 effects for the learning, teaching and assessment of English? Which criterial features can be used as diagnostics of proficiency at the individual learner level?

30 EP Research Programme How does learning to speak differ from learning to write? What determines communicative success and comprehensibility in these two language modes? What are the similarities and differences between adult and young learners of English developmentally and at each stage of learning? What is the role of learner and learning strategies? How do all the previous factors interact during language learning? How do they predict likely versus less likely patterns of learner output? What type of learning model can accommodate these multi-factor interactions that underpin language learning?

31 Key Concept - Criterial features These are properties of English - constructions, words, rule types, errors and their frequencies, etc. -that are distinctive and characteristic of L2 proficiency at one or more levels English Profile Studies series (Cambridge ESOL/CUP) Volume 1: Criterial Features in L2 English Hawkins & Filipovi, 2012 Also Hawkins & Buttery (2009, 2010) Hendriks and Saville (2009)

32 May 2012 With a preface by John Trim

33 English Profiles projects Practical outcomes related to: Vocabulary Grammar Functions

34 EP Profiles online with Handbooks English Vocabulary Profile (EVP) a vocabulary database by CEFR level now available English Grammar Profile (under development) a database of grammatical structures by CEFR level English Functions Profile (under development) a database of real English examples for various functions in different contexts by CEFR level Online resources accompanied by handbooks to guide users

35 English Profile Journal An incrementally-published, public domain journal The aim is to share the latest research resulting directly from the English Profile Programme and similar research EPJ publishes articles, plenary talks, notes, and reviews Editors: Michael McCarthy University of Nottingham, UK John Hawkins University of Cambridge, UK Available free

36 The EP Network Collaboration with colleagues working with learners in their own context Open to all to become involved depending on interests Regular events and seminars Annual EP Network Seminar in Cambridge

37 The EP Booklet (V1.1) Includes: How to get involved with EP The EP Network References Available on line

38 Automated Assessment: Background Cambridge ESOL: a global network nearly 4 million language tests each year in over 130 countries a global network of over 25,000 examiners logistics of marking exams, examiner training, examiner monitoring Proliferation of AA systems at least 12 programmes for AA writing, e.g. Educational Testing Service s erater PearsonKT s KAT Engine/Intelligent Essay Assessor

39 Automated Assessment - Writing Project led by Prof Ted Briscoe and team (Computer Lab) Cambridge Assessment PhD studentships: O.Andersen (completed) H.Yannakoudakis (ongoing) E.Kochmar (ongoing) Patented AA tool: Briscoe, E. J., Medlock, B., and Andersen, Ø. (2010). Automated Assessment of ESOL Free Text Examinations. University of Cambridge Technical Report, 790.

40 Automated Assessor Briscoe, Medlock & Andersen (2010) distinguishes Pass-Fail FCE exam scripts uses distinct, new methodology better suited to ESOL text: treats AA of ESOL text as rank preference learning performs very close to agreement levels achieved by human examiners

41 Discriminative features Automated Assessor exploits textual features for classification (i) Word n-grams (ii) Part-of-speech (POS) n-grams (iii)phrase structure (PS) rules (iv)error-rate extracts + and - discriminative linguistic features that contribute to classification Later versions Yannakoudakis, Briscoe & Medlock (2011) Yannakoudakis & Briscoe (2012) discourse coherence features

42 Discriminative features Feature Type Example Prediction VM_RR PoS bigram could clearly +,_because word bigram because of - necessary word unigram it is necessary that + the_people word bigram *the people are clever - VV0_VV0 PoS bigram *we go see film - NN2_VVG PoS bigram children smiling +

43 From computational linguistics and corpora to language assessment Validating a Writing Assessment Scale: Insights from SLA, Corpora, and Computational Linguistics (Salamoura & Lim, 2012)

44 Revised Cambridge First (FCE) scale Analytic Scale CEFR linked: B2

45 Scale Descriptors Grammar/Structure Descriptor B2: Uses a range of simple and some complex grammatical forms with a good degree of control

46 Scale Validation Quantitative Investigation Qualitative Investigation ? scale descriptors = text features

47 Visual User Interface Yannakoudakis, Briscoe & Alexopoulou (2012) Alexopoulou, Yannakoudakis & Salamoura (submitted) tool for exploration of corpus graphs & lists of features: by discriminative power, feature type, frequency, etc. correlations: feature-feature, feature-error, featuregrade, etc. links to relevant scripts & other meta-data

48

49 Grammar/Structure Descriptor B2: Uses a range of simple and some complex grammatical forms with a good degree of control

50 Features (from top 40 discriminative) Verb Phrase Complex word order: Modal verb + adverb + verb VM_RR_VV0 will clearly see, can always phone, should often advise Complex tense VH0_VVN_NN1 have done work VVN_TO wanted to, tried to VHG_VVN having given, After having spent Noun Phrase Complex nominal reference DD_JJ any good, some older NPD2 Sundays, Mondays VVN_NN1 the deserved award, guided tour NN2_VVG more people thinking that, looking at the boats arriving and going away

51 Reciprocal benefits Lg Assessment & Learner Corpora Lg Assessment Writing Scale NLP / AA for Writing NLP / Visualisation tool

52 The Assessment of Speaking Speaking tests for English for Speakers of Other Languages (ESOL) have a long association with Cambridge University, e.g. the Certificate of Proficiency in English (CPE) 1913 onwards Challenges: Defining the speaking construct Deciding how to assess it Human vs automated assessment

53 Sample collaborative task

54 Candidates are expected to: 1. Talk together about why two of the photographs might have been taken. 2.Talk about which of the pictures would have the most impact for a publicity campaign. Evidence of: exchanging ideas, expressing & justifying opinions, (dis)agreeing, suggesting, speculating, evaluating, negotiation, etc.

55 Defining the speaking construct Speaking involves multiple competencies, e.g. vocabulary and grammatical knowledge, phonological control, knowledge of discourse, and pragmatic awareness. Generally involves reciprocal oral interaction and tends to be assessed directly, through a face-to-face test. Speaking tests can be delivered and assessed by humans or through technology.

56 Project 1: Measuring L2 English phonological & prosodic proficiency Collaboration with Dr Brechtje Post and team (Department of Theoretical & Applied Linguistics) PhD students: Calbert Graham, Aike Li, Elaine Schmidt Prof Mike McCarthy (Universities of Nottingham, Limerick, Penn State) Drs Fiona Barker, Evelina Galaczi, Angeliki Salamoura, Nick Saville (Cambridge ESOL) (Galaczi, Post, Li & Graham 2011) (Barker, Post, Schmidt & McCarthy 2011)

57 Rationale & research questions Relatively little is known about the way in which phonological and prosodic proficiency is acquired and displayed across proficiency levels. How reliably can levels of pronunciation ability be discriminated in L2 speech? What pronunciation measures have the highest discriminative properties across proficiency levels?

58 Pilot study: data & measures 16 speech samples across 4 CEFR levels. Measures: rhythm metrics; sound segments and syllable structures; sentence stress and boundary marking. PRAAT speech processing software used to annotate samples prior to statistical analysis.

59 Pilot study: findings Levels of pronunciation ability can be objectively discriminated in L2 speech across levels (RQ1) Rhythm metrics seem to play a role as criterial distinguishing features between levels. Further work is planned to identify which measures provide the highest discriminative properties (RQ2) across an extended dataset and for further L1s.

60 Project 2: Automated Assessment of Speaking Collaboration with Prof Steve Young, Prof Phil Woodland, Dr Bill Byrne, Dr Mark Gales, Dr Pirros Tsiakoulis and team (Speech Group, Machine Intelligence Lab, Department of Engineering)

61 Rationale There has been considerable research into automatic evaluation of specific aspects of speech e.g. fluency or pronunciation. Automated scoring systems are currently in use: for scripted/highly constrained speech in tests for 1 minute samples of practice TOEFL tests We seek to develop a new way to automatically evaluate spoken test performance in an online domain-specific test.

62 Initial stage: model-building We aim to build an automatic speech recognition system and scoring system, using speaking test performances, scores and other spoken and written data across CEFR levels. Score predictions will be based on automatic speech recognition, NLP and machine learning approaches. These will be evaluated with regard to human scores. The system will also have the potential for transcribing speech.

63 Uses of Automated Assessment Range of scenarios: Low stakes: practice tests, formative feedback, placement purposes Higher stakes: live test administration, alongside/instead of human examiners, examiner monitoring Fitness for purpose

64 What s next for Cambridge English? Extend our collaborative research, sponsorship and other activities within the University. Find out more:

65 What s next for Cambridge English? Extend our collaborative research, sponsorship and other activities within the University. Find out more: Thank you