Developing Linguistic Descriptors for the Assessment of Advanced Proficiency with the

Transcription

1 Symposium on Writing Assessment in Higher Education: Making the framework work Free University of Amsterdam, October 2011 Developing Linguistic Descriptors for the Assessment of Advanced Proficiency with the Ekaterina Zaytseva & Marcus Callies Department of English and Linguistics Johannes-Gutenberg Universität Mainz, Germany

2 Roadmap 1. Introduction 2. Assessing advanced proficiency in academic writing 2.1 Assessing proficiency in SLA research 2.2 A corpus-based approach to assessing writing proficiency 3. The Corpus of Academic Learner English (CALE) 3.1 Design and composition 3.2 Annotation 3.3 Linguistic descriptors 4. Conclusion 2

3 1. Introduction: (Learner) Corpus linguistics (1) Corpus linguistics: "the study of language based on examples of 'real life' language use" (McEnery & Wilson 2001) Corpus: systematic collection of authentic, continuous and contextualized language use stored in electronic format Learner corpus: systematic collection of authentic, continuous and contextualized language use by foreign or second language learners, stored in electronic format McEnery & Wilson (2001)

4 1. Introduction: (Learner) Corpus linguistics (2) Learner corpora used in (European) SLA research for almost two decades, typically focusing on "advanced" L2 learners Learner Corpus Research (LCR) has contributed substantially to description of advanced interlanguages, producing evidence that - texts produced by ALs & NS differ in terms of frequencies of certain words, phrases, syntactic structures - learners often demonstrate unawareness of register differences Granger (2009)

5 1. Introduction: Advanced proficiency (1) Increasing interest in advanced stages of acquisition and questions of near-native competence in SLA research Advanced learners of various L1 backgrounds have similar problems & face similar challenges on their way to near- native proficiency: - optional &/or highly L2-specific phenomena, at interfaces of linguistic subfields (e.g. syntax-semantics, syntax-pragmatics) - academic writing: lack of understanding of conventions of academic writing &/or lack of practice Advanced Learner Varieties (ALVs) DeKeyser (2005), McCrostie (2008)

6 1. Introduction: Advanced proficiency (2) still no set of well-defined criteria to classify such learners as "advanced" or even "near-native esp. in case of very advanced learners - impressionistic judgment without further testing desiderata: - definition & clarification of "advancedness" - in-depth description of advanced learner language (corpus-based) - operationalization of such description, using linguistic criteria for the assessment of advancedness Coppieters (1987), Leube (2000)

7 2.1 Assessing proficiency in SLA research (1) Effects of varying levels of proficiency: little attention in SLA research! L2 proficiency often inadequately assessed, limiting generalizability of research results 2 tendencies concerning L2 proficiency assessment (Thomas 2006): - increased variety & diversity of L2 proficiency assessment - tendency to disregard importance of L2 proficiency assessment Ortega & Byrnes (2008) discuss four (partially overlapping) global measures commonly used to operationalize advancedness: - institutional status - standardized tests - late-aquired features of language - sophisticated language use in context (e.g. choice among registers, repertoires, and voice) basis for development of descriptors characteristic of academic prose Thomas (1994, 2006); Ortega & Byrnes (2008)

8 2.1 Assessing proficiency in SLA research (2) Writing proficiency: - expert raters - writing tasks (as part of tests) - rating degree - rating scale A level in Common European Framework of Reference (CEFR) Problems with levels in CEFR: - descriptors & can-do-statements illustrative, functional - too global & underspecified

9 2.2. A corpus-based approach to assessing writing proficiency (1) increasing awareness of need to develop linguistic descriptors ( "criterial features") desideratum: operationalization of (corpus-based) linguistic description of advanced learner language for language assessment learner corpora increasingly used for language testing and assessment (e.g. to inform what can be/is tested at a particular level) learner corpora have potential to increase transparency, consistency & comparability in assessment of L2 proficiency Taylor & Barker (2008), Barker (2010), Hawkins & Buttery (2010), Neff & Bunce (2011)

10 2.2. A corpus-based approach to assessing writing proficiency (2) BUT: several problems with existing learner corpora (e.g. ICLE): no corpora of "real" academic writing (yet) often insufficient control of variables, such as task-instruction, imagined audience, and register/genre proficiency level a fuzzy variable: - globally assessed by means of external criteria, i.e. institutional status - proficiency level varies between higher intermediate to advanced - "across-the-board" problem Granger et al. (2009)

11 2.2. A corpus-based approach to assessing writing proficiency (3) Results of human rating of 20 essays per ICLE-subcorpus acc. to CEFR levels (Granger et al. 2009:12)

12 The (CALE)

13 3.1 CALE: Design and composition (1) CALE: Language for Specific Purposes learner corpus, containing discipline- and genre-specific texts focus on written academic English ("academic learner writing") existing learner corpora (e.g. International Corpus of Learner English) largely include general argumentative/literary writing, not academic writing several typical features of academic writing not represented (frequently enough) Native-speaker control corpora (L1 writers of similar academic standing) - Michigan Corpus of Upper-Level Student Papers (MICUSP) - British Academic Written English corpus (BAWE) Currently mainly collecting data from German university students of English (some Chinese & Portuguese L1 available) Data from EFL learners of other L1 backgrounds needed for crosslinguistic and typological comparisons as to potential L1 influence Alsop & Nesi (2009), Granger et al. (2009), Granger & Paquot (forthc.), Römer & Brook O'Donnell (in prep.)

14 3.1 CALE: Design and composition (2) seven academic text types ("genres") produced as assignments in university courses

15 3.1 CALE: Design and composition (3) text classification comparable with NS control corpora, but clear(er) textual profiles, adopting situational characteristics & linguistic features identified for academic prose by Biber & Conrad (2009) Main classifying principle: communicative purpose/goal of text, genres set apart in terms of a. text's general purpose b. its specific purpose(s) c. the skills the author demonstrates d. the author's stance In addition, listing of major features of each text type as to a. structural features b. length c. functional features Biber & Conrad (2009)

16 Example: Abstract (ABS) communicative goal/purpose a. general purpose b. specific purpose(s) c. skills d. stance a. informational inform b. captures essence of published research (why, how, what: research focus, methodology results/findings, conclusion & recommendations IMRD); should help reader to quickly ascertain purpose, content and usefulness of publication c. author demonstrates ability to extract and provide essential information in an exhaustive and compelling way d. author s opinion/evaluation absent features a. structural b. length C. functional a. not structured into sections; appears at beginning of text it accompanies; may also occur as stand-alone entity instead of full paper b. rather short (approx words), rarely exceeding 500 words c. self-contained piece of writing, can be understood independently from accompanying publication

17 3.2 CALE: Annotation (1): 5 steps 1. participants submit texts electronically (usually in.doc/x or.pdf) manual pre-processing 2. extensive "non-linguistic" information (TOCs, lists of references, tables, figures, etc.) deleted, substituted by placeholder tags around headings/captions 3. body of text annotated for meta-textual data, i.e. underlying structural features (section titles, paragraphs, quotations, examples, etc.) annotation tools (UAM corpus tool, Pacx) 4. texts annotated (in file header) for metadata, i.e. learner variables (e.g. L1, age, gender, etc.) collected through written questionnaire

18 Learner profile questionnaire (excerpt)

19 3.2 CALE: Annotation (1): 5 steps 1. participants submit texts electronically (usually in.doc/x or.pdf) manual pre-processing 2. extensive "non-linguistic" information (TOCs, lists of references, tables, figures, etc.) deleted, substituted by placeholder tags around headings/captions 3. body of text annotated for meta-textual data, i.e. underlying structural features (section titles, paragraphs, quotations, examples, etc.) annotation tools (UAM corpus tool, Pacx) 4. texts annotated (in file header) for metadata, i.e. learner variables (e.g. L1, age, gender, etc.) collected through written questionnaire 5. file header also includes set of further metadata that pertain to each individual text (genre, type of course, discipline, setting etc.) also collected with questionnaire

20 Text profile questionnaire (excerpt)

21 3.2 CALE: Annotation (2) further levels of annotation based on function-to-form approach to analysis of learner language example: annotating rhetorical functions (like contrast or exemplification) and lexico-grammatical means to express them advantages of function-oriented levels of annotation: - learners = active language users who do not always/necessarily demonstrate deficient knowledge of L2 (may use non-canonical, strategic means to express communicative functions) - makes possible identification & documentation of near-complete inventory of lexical-grammatical means used to express various communicative functions in written discourse

22 3.3 CALE: Linguistic descriptors (1) operationalization of (corpus-based) linguistic description of advanced learner language, using linguistic criteria for language assessment Develop set of linguistic criteria characteristic of academic prose for assessing advanced proficiency in general & proficiency in academic writing, based on CALE & survey of existing research literature Which are these? often optional / highly L2-specific features, located at interfaces of linguistic subfields Biber & Conrad (2009), Callies (2010), Callies & Zaytseva (2011), Gilquin (to appear), Wulff & Gries (2011)

23 3.3 CALE: Linguistic descriptors (2) Candidates for "positive linguistic properties : 1. use of specific constructions (verb-argument constructions, focus constructions, raising, mediopassive) 2. use of inanimate subjects (This paper discusses, The results suggest that ) 3. phrases to express rhetorical functions (e.g. by contrast, to conclude, In sum) 4. typical lexical co-occurrence patterns (e.g. conduct, carry out, undertake as typical verbal collocates of experiment, analysis, research) 5. varied repertoire of reporting verbs (discuss, claim, suggest, argue etc.) Biber & Conrad (2009), Callies (2010), Callies & Zaytseva (2011)

24 From external criteria to linguistic descriptors global measure pool of learners contributing texts to corpus; external measure: institutional status local measures clustering techniques using linguistic descriptors E.g. use of reporting verbs <diverse (e.g. say, state) >diverse (e.g. claim, discuss, argue, etc.)

25 4. Conclusion CALE: learner corpus; contains discipline- and genre-specific texts with the focus on advanced learner academic writing CALE-based research linguistic descriptors anchored in academic writing for measuring / assessing advanced proficiency (in academic writing)

26 References (1) Ädel, A. (2006), Metadiscourse in L1 and L2 English. Amsterdam: Benjamins. Ädel, A. (2008), "Involvement features in writing: do time and interaction trump register awareness?", in Gilquin, G., M.B. Diez- Bedmar & S. Papp (eds.), Linking up Contrastive and Learner Corpus Research. Amsterdam: Rodopi, Alsop, S. & H. Nesi (2009), "Issues in the development of the British Academic Written English (BAWE) corpus", Corpora 4(1), Barker, F. (2010), "How can corpora be used in language testing?" in Anne O'Keeffe & Michael McCarthy (eds.), The Routledge Handbook of Corpus Linguistics. New York: Routledge, Biber, D. & S. Conrad (2009), Register, Genre, and Style. Cambridge: CUP. Callies, M. (2010), The (non-)representation of authorship in L2 academic writing. Paper presented at ICAME 31 "Corpus Linguistics and Variation in English", May 2010, Giessen/Germany. Callies, M. (2008), "Easy to understand but difficult to use? Raising constructions and information packaging in the advanced learner variety", in Gilquin, G., M.B. Diez-Bedmar & S. Papp (eds.), Linking Up Contrastive and Learner Corpus Research. Amsterdam: Rodopi, Callies, M. & K. Szczesniak (2008), "Argument realization, information status and syntactic weight - A learner-corpus study of the dative alternation", in Grommes, P. & M. Walter (eds.), Fortgeschrittene Lernervarietäten. Korpuslinguistik und Zweitspracherwerbsforschung. Tübingen: Niemeyer, Callies, M. & E. Zaytseva (2011), "The Corpus of Academic Learner English (CALE): A new resource for the study of lexicogrammatical variation in advanced learner varieties," in Hedeland, H., T. Schmidt & K. Wörner (eds.), Multilingual Resources and Multilingual Applications (Hamburg Working Papers in Multilingualism B 96), Coppieters, R, (1987), "Competence differences between native and near-native speakers", Language 63, DeKeyser, R. (2005), "What makes learning second language grammar difficult? A review of issues", Language Learning, 55(s1), Gilquin, G. (to appear), "Lexical infelicity in causative constructions. Comparing native and learner collostructions", in Leino, J. & R. von Waldenfels (eds.), Analytical Causatives, München: Lincom Europa. Gilquin, G. & M. Paquot (2008), "Too chatty: Learner academic writing and register variation", English Text Construction 1(1), Granger, S. (2009), "The contribution of learner corpora to second language acquisition and foreign language teaching: A critical evaluation", in Aijmer, Karin (ed.), Corpora and Language Teaching. Amsterdam: Benjamins, Granger, S., E. Dagneaux, F. Meunier & M. Paquot (2009), The International Corpus of Learner English. Version 2. Handbook and CD-ROM. Louvain-la-Neuve: Presses Universitaires de Louvain. Granger, S. & M. Paquot (forthcoming), "Language for Specific Purposes Learner Corpora", in Upton, T.A. & Connor, U. (eds.), Language for Specific Purposes. The Encyclopedia of Applied Linguistics. New York: Blackwell. Gries, S. Th. & S. Wulff (2005), "Do foreign language learners also have constructions? Evidence from priming, sorting, and corpora", Annual Review of Cognitive Linguistics 3,

27 References (2) Gries, S. Th. & S. Wulff (2009), "Psycholinguistic and corpus-linguistic evidence for L2 constructions", Annual Review of Cognitive Linguistics 7, Gries, S. Th. & S. Wulff (2011). Constituent-order alternation phenomena in L2: two multifactorial and processing-based case studies. Paper presented at Learner Corpus Research 2011, University of Louvain/Belgium, September Hawkins, J. & P. Buttery (2010), "Criterial features in learner corpora: theory and illustrations," English Profile Journal 1(1), Leech, G. (2011), "Frequency, corpora and language learning," in DeCock, S. et al. (eds.), A Taste for Corpora. Amsterdam: Benjamins, Leube, K. (2000). Information Structure and Word Order in the Advanced Learner Variety. Hamburg: bod.libri. McCrostie, J. (2008), "Writer visibility in EFL learner academic writing: A corpus-based study", ICAME Journal 32, McEnery, A.M. & Wilson, A. Corpus Linguistics (second edition). Edinburgh: Edinburgh University Press. Neff van Aertselaer, J. & C. Bunce (2011), "The use of small corpora for tracing the development of academic literacies," in DeCock, S. et al. (eds.), A Taste for Corpora. Amsterdam: Benjamins, Ortega, L. & H. Byrnes (2008), "The longitudinal study of advanced L2 capacities: An introduction", in Ortega, Lourdes & Heidi Byrnes (eds.), The Longitudinal Study of Advanced L2 Capacities. New York: Routledge/Taylor & Francis. Römer, U. & M. Brook O Donnell (in preparation). From student hard drive to web corpus: The design, compilation, annotation and online distribution of MICUSP. Taylor, L. & F. Barker (2008), "Using corpora for language assessment", in Shohamy, E. & N.H. Hornberger (eds.), Encyclopedia of Language and Education, 2nd Edition, Volume 7: Language Testing and Assessment. New York: Springer, Thomas, M. (1994), "Assessment of L2 proficiency in second language acquisition research", Language Learning 44:2, Thomas, M. (2006), "Research synthesis and historiography: The case of assessment of second language proficiency", in Norris, J.M. & L. Ortega (eds.), Synthesizing Research on Language Learning and Teaching. Amsterdam: Benjamins, Wulff, S. & S. Th. Gries (2011), "Corpus-driven methods for assessing accuracy in learner production", in Robinson, P. (ed.), Second Language Task Complexity: researching the Cognition Hypothesis of language learning and performance. Amsterdam: Benjamins,