A Verbal Protocol Analysis of Test-takers Processes of Taking IELTS Academic Reading Test 雅思学术类阅读测试受试答题过程有声思维研究

Transcription

1 A Verbal Protocol Analysis of Test-takers Processes of Taking IELTS Academic Reading Test 雅思学术类阅读测试受试答题过程有声思维研究 Xiangdong Gu Qin Lv Lu Zhao Chongqing University 6 th Nov. 2013

2 Our Teams Studies Using Verbal Protocol Analysis Gao. X & Gu. X (2008) An introspective study on test-taking process for banked cloze, Teaching English in China (4), Zheng. Y & Gu. X (2009) A study on the construct validity of long dialogue multiple-choice listening comprehension tests: retrospection approach, Teaching English in China (6), Gu. X & He.Y (2011) An introspective study on testees process of taking English to Chinese translation test in TEM 8, Foreign Languages and Translation (3), Gu. X & Shi. C (2012) A retrospective study on test-takers cognitive and metacognitive processes in taking a compound dictation test, Chinese Journal of Applied Linguistics 35 (4), Gu. X & Wang. J (2012) An introspective study on test-takers processes of taking sentence translation test in CET-4 Gu. X & Shen. Y (2012) An introspective study on test-takers processes of taking vocabulary test in reading section of HSK-6 Gu, X, Lv. Q & Zhao. L (2013) A verbal protocol analysis of test-takers processes of taking IELTS academic reading test

3 Examples of IELTS Studies Vol 4 The Development of IELTS: A study of the effect of background knowledge on reading comprehension --Caroline Clapham (1996) Vol 19 IELTS Collected Papers: Research in speaking and writing assessment --Edited by Taylor & Falvey (2007) Vol 23 Assessing Academic English: Testing English proficiency, the IELTS solution --Alan Davies (2008) Vol 24 Impact Theory and Practice: Studies of the IELTS test and Progetto Lingue Roger Hawkey (2006) Vol 25 IELTS Washback in Context: Preparation for academic writing in higher education --Anthony Green (2007) Vol 34 IELTS Collected Papers 2: Research in reading and listening assessment --Edited by Lynda Taylor and Cyril J Weir (2012)

4 Academic Reading Weir. C, Hawkey. R, Green. A, Ünaldi. A & Devi. S Moore. T, Morton. J & Price. S Weir. C, Hawkey. R, Green. A & Devi. S Green. A & Hawkey. R The relationship between the academic reading construct as measured by IELTS and the reading experience of students in their first year of study at a British university Construct validity in the IELTS Academic Reading test: a comparison of reading requirement in IELTS test items and in university study The cognitive processes underlying the academic reading construct as measured by IELTS An empirical investigation of the process of writing Academic Reading test items for the International English language

5 Definition of Verbal Protocols Rich data sources containing individuals spoken thoughts that are associated with working on a task. Subjects usually either think aloud as thoughts occur to them while working on a particular task (concurrent introspection) or verbalize their thoughts upon completion of the task (immediate retrospection). The verbalizations are recorded and then coded according to theory-driven and/or empirically driven categories. (Gale Encyclopedia of Education: Research methods) Source:

6 Validity and Reliability of Verbal Protocols Information captured within verbal reports Validity corresponds closely with Information actually needed similar verbal protocols Subject 1 of the same task Subject 1 Coder 1 Coder 2 Reliability coding scheme accurately captures the behavior Coder 1 Coder 2

7 Research Questions What types of strategies do test-takers employ when taking IELTS academic reading test? Are there any similarities and differences in the employment of strategies between high-level group and low-level group of testtakers? Are there any similarities and differences about test-takers employment of strategies among different task types?

8 Task Types All task types of IELTS Identifying information (True/False/Not given) Matching information Sentence completion Diagram label completion Short-answer questions Identifying writer s views/claims (Yes/No/Not given) Matching headings Matching features Matching sentence endings Multiple choice Summary/note/table/flow chart completion Task types in selected materials 1 Summary/Note/Table/Flow-chart completion 2 Identifying information (True/False/Not given) 3 Matching headings 4 Summary/Note/Table/Flow-chart completion 5 Matching features 6 Matching headings 7 Multiple choice 8 Sentence completion

9 Instrument: Test 2 in Cambridge IELTS Test 8 (2011) Passage 1 Passage 2 Passage 3 Topic Sheet glass manufacture the float process The Little Ice Age The meaning and power of smell Section 1 Tasks Summary/Note/Table/ Chart-flow completion Matching headings Matching headings Section 2 Tasks Identifying information (True/False/Not given) Summary/Note/Table/ Chart-flow completion Multiple choice Section 3 Matching features Sentence completion Tasks Item No

10 Operational Procedures Task identification and analysis Selecting an appropriate procedure Selecting and Training subjects Collecting verbal reports and supplementary data Transcribing verbal reports Developing an encoding scheme Segmenting and encoding protocols Calculating encoder reliability Analyzing data decision on whether to use verbal protocol analysis motivated, skilled at verbalization, different proficiency levels, pilot, rehearsal simultaneous introspection, immediate retrospection; subject performance first, thinking aloud second; researcher out of subjects view; retrospective interview; first language; clear tape or video recording word by word, time markers, double check units division, intra- or/and inter- reliability descriptive statistics, Chi-square Tests Inter-rater Coding Agreement of this study is 0.78.

11 Background Information of the Participants Participants Gender Age Major Education background IELTS score SL1 Male 24 Mechanics PhD candidate 6.0 SL2 Male 26 Power engineering MS candidate 5.5 SL3 Female 22 Measurement and control MS candidate 5.0 SH4 Male 25 Electrical engineer PhD candidate 8.0 SH5 Female 23 Chinese linguistics and literature MA candidate 7.5 SH6 Female 24 Law MA candidate 7.0 SL: Subject of Low-level; SH: Subject of High-level

12 Abbott s Reading Strategy Classification (2010) Bottom-up strategies B1 B2 B3 B4 B5 B6 B7 Breaking lexical items into parts Scanning for explicit information requested in the item Identifying a synonym or a paraphrase of the literal meaning of a word, phrase, or sentence Relating visual or verbal information to accompanying visuals Matching key vocabulary in the item to key vocabulary in the text Using knowledge of grammar or punctuation Using local context cues to interpret a word or phrase Definition Breaking words into smaller units to promote comprehension Scanning the text for specific details or explicit stated information requested in the item Identifying or formulating a synonym or a paraphrase of the literal meaning of a word, phrase, or sentence in the text to help answer the question Matching visual information in the item to answer the question Matching visual information in the item to the words in the text to answer the question Using awareness of grammar, syntax, parts of speech, or punctuation to help answer the question Using the words in a sentence that precede or follow a specific word or phrase to understand a particular word or phrase

13 Abbott s Reading Strategy Classification (2010) Top-down strategies Definition T1 T2 T3 T4 T5 Skimming for gist or identifying the main idea, theme, or concept Connecting or relating information presented in different sentences or parts of the text Drawing an inference based on information presented in the text Speculating beyond the text Recognizing discourse format Drawing on the major points of the passage to answer the question; summarizing main concept Relating new information to previously stated information to help answer the question; synthesizing scattered information Making an inference, drawing a conclusion, or forming a hypothesis based on information not explicitly stated in the text to answer the question Using background information to speculate beyond the text Using discourse format or text organization to answer the question (eg., discriminates between fact and opinion or cause and effect; or notes how the information is presented)

14 Bottom-up strategies Coding Schema of Bottom-up Strategy Use B1 Breaking lexical items into parts / B2 B3 B4 B5 Scanning for explicit information requested in the item Identifying a synonym or a paraphrase of the literal meaning of a word, phrase, or sentence Relating visual or verbal information to accompanying visuals Matching key vocabulary in the item to key vocabulary in the text Examples 这有一段话, 让选词,weather during in the distant are 什么, 应该是 tree rings and ice cores, tree rings, ice cores, 这部分在 paragraph 3 里面找的到,Records tree rings and ice cores( 第三段的原文 ) (Participant 1) A 段的标题, 我觉得它大多数都在讲 emo tion 有一个 feelings 很像, 应该算是这个 emotion 的近义词吧, 所以选 (Participant 5) 先看这个图 这有两个轮子, 刚好第二段讲到这个问题, 估计就应该填 roller, 因为文中提到两个 two hot rollers (Participant 6) 重新再看一下这个文章, 找这几个关键词, 第一个是 discover all the lands 那就逐行看, 找关键词吧 (Participant 1) B6 B7 Using knowledge of grammar or punctuation Using local context cues to interpret a word or phrase however, 它这个呢就是一个转折, 前面是优点, 后面肯定就是讲的一个缺点 所以就可以直接做题了 (Participant 1) /

15 Top-down strategies T1 T2 Coding Schema of Top-down Strategy Use Skimming for gist or identifying the main idea, theme, or concept Connecting or relating information presented in different sentences or parts of the text Examples ( 读文章 )Global temperatures 这一段主要讲的是人类对温度的这样一个影响, 恩, 所以就是选这个四, human impact on the climate. (Participant 3) / T3 T4 T5 Drawing an inference based on information presented in the text Speculating beyond the text Recognizing discourse format 12 题,the process improved, 有没有说提到 improved 呢? 就是说, ( 在文中找相应信息 ) they finally succeeded 15 years, 就是说之前的时候它要停, 而且要校正, 需要花费四个月, 所以这个时候呢我们推断一下, 应该是 improved. (Participant 4) 9 到 13 是一个判断问题 true or false 的问题, 按照雅思的一个规则, 那它就不会在第一部分出现 (Participant 3) 前面讲到了优点, 再看, 后边应该就是缺点了吧,( 继续看文章 ),glass was marked, 恩, 第五个空应该就是填 marked. (Participant 6)

16 Frequency of Participants Bottom-up Strategy Use Subject B1 B2 B3 B4 B5 B6 B7 SL SL SL SH SH SH

17 Frequency of Participants Bottom-up Strategy Use B2: Scanning for explicit information requested in the item

18 Frequency of Participants Top-down Strategy Use Subject T1 T2 T3 T4 T5 SL SL SL SH SH SH

19 Frequency of Participants Top-down Strategy Use T1: Skimming for gist or identifying the main idea, theme, or concept T3: Drawing an inference based on information presented in the text T5: Recognizing discourse format

20 Frequency of All Participants Strategy Use Chi-square tests: Sig..001

21 Frequency of Participants Bottom-up Strategy Use in Each Task Type Task Strategy B1 B2 B3 B4 B5 B6 B7 Task 1 Task 2 Summary/Note/Table/Flow-chart completion Identifying information (True/False/Not Given) Task 3 Matching headings Task 4 Summary/Note/Table/Flow-chart completion Task 5 Matching features Task 6 Matching headings Task 7 Multiple choice Task 8 Sentence completion

22 Frequency of Participants Top-down Strategy Use in Each Task Type Task Strategy T1 T2 T3 T4 T5 Task 1 Task 2 Summary/Note/Table/Flow-chart completion Identifying information (True/False/Not Given) Task 3 Matching headings Task 4 Summary/Note/Table/Flow-chart completion Task 5 Matching features Task 6 Matching headings Task 7 Multiple choice Task 8 Sentence completion

23 Bottom-up strategies more frequently used in Tasks 1, 4, 5 & Summary/Note/Table/Flow-chart completion Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Bottom-up Strategies Matching features Top-down Strategies Sentence completion

24 Top-down strategies more frequently used in Tasks 3 & Matching headings Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Bottom-up Strategies Top-down Strategies

25 Bottom-up and top-down strategies almost equally used in Tasks 2 & Identifying information (True/False/Not Given) Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Bottom-up Strategies Top-down Strategies Multiple choice

26 Conclusions Q1: What types of strategies do test-takers employ when taking IELTS academic reading test? Test-takers cognitive processes of taking IELTS academic reading test are very complex. They employ both bottom-up strategies and top-down strategies when taking the test. However, some strategies (B2, B5, T1 & T3) are much more frequently used while some strategies (B1, B7 & T2) are not used at all.

27 Conclusions Q2: Are there any similarities and differences in the employment of strategies between high-level group and low-level group of test-takers? Both high-level and low-level groups of test-takers tend to use bottom-up strategies more frequently and there is no significant difference in the frequency of their bottom-up strategy use. But high-level group use more top-down strategies than low-level group with significant difference.

28 Conclusions Q3: Are there any similarities and differences about test-takers employment of strategies among different task types? In different tasks, test-takers strategy use is different. Bottomup strategies are more frequently used in three task types: Summary/Note/Table/Flow-chart completion, Matching features, Sentence completion; Top-down strategies are more frequently used in Matching features; Bottom-up and Top-down strategies are almost equally used in tasks of Identifying information (True/False/Not Given) and Multiple choice.

29 Implications A wider range of strategies need to be covered. The proportion of the tasks which measure test-takers top-down strategies needs to be increased as they can differentiate high-level group from low level group. More training and practice is needed for test-takers to improve their top-down strategy use.

30 Significance Theoretically, provided evidence to confirm the rationality of the construct framework. Practically, put forward suggestions for language testers on test development and test evaluation, and provided implications for language teachers and learners on how to improve strategy use. Methodologically, proved verbal protocol analysis as an effective approach to examine strategy use employed by test-takers.

31 Limitations Some subjects may be unrepresentative due to convenience sampling. The validity and reliability of the coding schemes need to be improved. The theoretical framework used for the analysis is inadequate, e.g. test-wiseness strategies are not included in it. The generalizability of the research findings remains to be further testified by triangulation methodology.

32 References and Further Readings 1. Abbott, M. L. (2010). An introspective study of Arabic and Mandarin speaker reading comprehension strategies. TESL Canada Journal, Anderson, N. C. (1991). Individual differences in strategy use in second language reading and testing. Modern Language Journal, 75(4): Alderson, J. C. (2000). Assessing Reading. Cambridge University Press, Cambridge. 4. Alderson, J. C., Clapham, C. & Wall, D. (2000). Language Test Construction and Evaluation. Beijing: Foreign Language Teaching and Research Press. 5. Barkaoui, K. (2011). Think-aloud protocols in research on essay rating: An empirical study of their veridicality and reactivity. Language Testing, 28(1), Carver, R. (1992). Reading Rate: theory, research and practical implications. Journal of Reading, 36(2), Cohen, A. (1984). On taking language tests: What the students report, Language Testing, 1, Cumming, A. (1990). Metalinguistic and ideational thinking in second language composing. Written Communication 7, Enright, M., Grabe, W., Koda, K., Mosenthal, P., Mulcany-Ernt, P. & Schedl, M. (2000). TOEFL 2000 reading framework: A working paper. TOEFL Monograph Series 17 ETS, Princeton. 10. Ericsson, K. A. & Simon, H. (1993). Protocol Analysis: Verbal Reports as Data. Cambridge, Massachusetts: The MIT Press. 11. Faerch, C. & Kasper, G. (1987). Introspection in second language research. Clevedon, Philadelphia: Multilingual Matters. 12. Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press. 13. Goodman, K. S. (1973). Psycholinguistic universals of the reading process, In E Smith (ed.), Psycholinguistics and Reading. New York: Holt Rinehart and Winston. 14. Grabe, W. & Stoller, F. L. (2002). Teaching and Researching Reading. Harlow, England: Pearson Education. 15. Glaser, B. G. (1978). Theoretical sensitivity: Advances in the methodology of grounded theory. Mill Valley, CA: Sociology Press. 16. Glaser, B. G. (1992). Basics of grounded theory analysis: Emergence vs. forcing. Mill Valley, CA: Sociology Press. 17. Grellet, F. (1981). Developing Reading Skills. Cambridge: CUP. 18. Hamp-Lyons, L. (1990). Second Language writing: assessment issues. In Kroll, B. (Eds.), Second Language Writing: Research Insights for the Classroom. Cambridge University Press, Cambridge,

33 References and Further Readings 19. Hammadou, J. (1991). Interrelationships among prior knowledge, inference, and language proficiency in foreign language reading. Modern Language Journal, 75, Hawkey, R. (2006). Impact theory and practice: Studies of the IELTS test and Progetto Lingue Cambridge University Press, Cambridge. 21. Hill, C., & Parry, K. (1992). The test at the gate: Models of literacy in reading assessment. TESOL Quarterly, 26, Hudson, T. & Park, S. (2002). Validity issues for selected versus constructed response Internet-based language test. Paper presented at AAAL, Arlington, Virginia. 23. IELTS (1996). The IELTS handbook. Cambridge: UCLES. The British Council, IDP Education Australia. 24. IELTS (2007). IELTS official practice materials. Cambridge University Press, Cambridge. 25. IELTS Handbook. (1999). Cambridge: University of Cambridge Local Examinations Syndicate. 26. IELTS Handbook. (2007). Cambridge: University of Cambridge Local Examinations Syndicate. 27. Irwin, J. W. (1986). Teaching Reading Comprehension Process. New Jersey: Prentice Hall Inc. 28. Khalifa, H. & Weir, C. (2009). Examining reading: Research and practice in assessing second language reading. Studies in Language Testing 29. Cambridge: UCLES & Cambridge University Press. 29. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, McDonough, S. (1995), Strategy and Skill in Learning a Foreign Language, Arnold. 31. McGinnis, D. J. & Smith, D. E. (1982). Analyzing and Treating Reading Problems. New York: Macmillan Publishing Co., Inc. 32. McNamara, T. F. (1999). Computer-adaptive testing: A view from outside. In Chaloub-Deville M. (Eds.), Issues in computeradaptive testing of reading proficiency. Cambridge University Press, Cambridge, Moore, T. & Morton, J. (1999). Authenticity in the IELTS academic module writing test: a comparative study of Task 2 items and university assignments in Tulloh, R. (Eds.), IELTS research reports 2. IELTS Australia, Canberra, Munby, J. L. (1978). Communicative Syllabus Design. Cambridge: CUP. 35. Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6, Nunan, D. (1995). Teaching methodology: a textbook for teachers. Phoenix ELT.

34 References and Further Readings 37. Oxford, R. (1990). Language learning strategies. New York: Newbury House. 38. Orr, M. (2002). The FCE speaking test: Using rater reports to help interpret test scores. System, 30, Parry, K. (1996). Culture, literacy and L2 reading. TESOL Quarterly, 30, Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing,1, Purpura, J. (1997). An analysis of the relationships between test takers cognitive and metacognitive strategy use and second language test performance. Language Learning, 47, Rennie, D. (1984). Clients tape-assisted recall of psychotherapy: A qualitative analysis. Paper presented at the annual meeting of the Canadian Psychological Association, Ottawa. 43. Richards, J. C., Plate, J. & Platte, H. (2000). Longman Dictionary of Language Teaching and Applied Linguistics. Beijing: Foreign Language Teaching and Research Press. 44. Rumelhart, D. E. (1980). Schemata:The building blocks of cognition. In Spiro, R. J., Bruce, B. C. & Brewer, W. E. (Eds.), Theoretical Issues in Reading Comprehension. Hillsdale, N. J. Lawrence Erlbaum. 45. Rupp, A. A., Ferne, T. & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: a cognitive processing perspective. Language Testing, 4, Schueller, J. (2004). Gender and foreign language reading comprehension: The effects of strategy training. Southern Journal of Linguistics, 27(1), Shi, C. Y. (2010). A Retrospective Study on Test-takers Processes of Taking Compound Dictation Test. Master of Arts dissertation, Chongqing University. 48. Taylor, G. (2009). A student s writing guide, Cambridge University Press, Cambridge. 49. Urquhart, A. & Weir, C. J. (1998). Reading in a second language: process, product and practice. Longman, London. 50. Vaughan, C. (1991). Holistic assessment: what goes on in the rater's mind? In Hamp-Lyons, L. (Eds.), Assessing Second Language Writing in Academic Contexts, Norwood, NJ: Ablex. 51. Wang, H. Z. (2009). Introspection in L2 speaking performance assessment research (1):12-22/127 CELEA Journal (Bimonthly). 52. Weir, C. J. (1983). Identifying the language needs of overseas students in tertiary education in the United Kingdom, PhD thesis, Institute of Education, University of London.

35 References and Further Readings 53. Weir, C. J., Yang, H. & Jin, Y. (2000). An empirical investigation of the componentiality of L2 reading in English for Academic Purposes, Studies in Language Testing, 12, UCLES Cambridge University Press, Cambridge. 54. Weir, C. J., Hawkey, R., Green, A., Devi, S. & Unaldi, A. (2009). The relationship between the academic reading construct as measured by IELTS and the reading experiences of students in the first year of their courses at a British university, IELTS Research Project Report. 55. Weir, C.J., Hawkey, R., Green, A. & Devi, S. (2009). The cognitive processes underlying the academic reading construct as measured by IELTS, in Thompson P. (Eds.), Research Reports Volume 9, British Council/IDP Australia, London. 56. Widdowson, H. G. (1978). Teaching Language as Communication. Oxford: OUP. 57. Wu, Y. A. (1998). What do tests of listening comprehension test?-a retrospection study of EFL test-takers performing a multiplechoice task. Language Testing 15 (1), Xiang dong, Gu & Xiao ying, Gao An introspective study on test-taking process for banked Cloze. CELEA Journal, 31(4), Yamashita, J. (2003). Processes of taking a gap-filling test: Comparison of skilled and less skilled EFL readers. Language Testing, 20, Yang, H. & Weir, C. (1998). Validation study of the national college English test. Shanghai: Shanghai Foreign Language Education Press. 61. Zhang, Y. (2006). Validation Study of Translation Items in National Entrance Test of English for MA/MS Candidates. Master of Arts dissertation, Wuhan University of Technology. 62. 陈晓扣, 李绍山 TEM-4 完型填空测试结构效度研究 答题过程分析法 [J]. 现代外语 (1): 金艳, 吴江 以 内省法 检验 CET 阅读理解测试的效度 [J]. 外语界 (2). 64. 郭纯洁 (2007). 有声思维法. 北京 : 外语教学与研究出版社. 65. 刘建达, 杨满珍 (2001). 成段改错试题考了什么?. 现代外语. 第 2 期 : 苗菊 (2005). 有声思维 翻译内在过程探索 [J], 外语与外语教学. 第 6 期 : 文军 (2005). 翻译课程模式研究 以发展翻译能力为中心的方法. 北京 : 中国文史出版社. 68. 文军, 孙三军 (2006). 论使用出声思维研究翻译过程. 外语学刊. 第 3 期 : 郑宇静, 辜向东 (2009). 长对话多项选择听力测试题结构效度的追述法研究 ( 英文 ). 中国英语教学. 第 6 期 :15-26.

36 Organizers Acknowledgements My MA team My family

37 Thank you very much for your attention! Xiangdong Gu, Qin Lv, Lu Zhao,