Lisbeth M Brevik. Mapping Tests: Comparing Upper Secondary Students Reading Skills in L1 and L2

Transcription

1 Mapping Tests: Comparing Upper Secondary Students Reading Skills in L1 and L2

2 Background Nationally distributed mapping tests L1: reading literacy in Norwegian (paper) L2: reading literacy in English (digital) Upper secondary school first year (15-16 years old) Same population as PISA 2012 PISA in April-May (4.686 students) Mapping tests in September ( students)

3 Sample L1-L2 Norwegian (L1): merged: English (L2):

4 Crosslinguistic reading L1-L2 L1 L2

5 Method Secondary data collection L2 from Norwegian Directorate L1 from schools & counties Quantitative analysis (SPSS) Merging the two datasets Reliability analysis Factor analysis Regression analysis

6 Research Questions 1. To what extent can reading comprehension in English (L2) among first year upper secondary students be explained by reading literacy in Norwegian (L1), and language knowledge in English (L2)? 2. What insights about L2 reading literacy can be derived from the 20% lowest performers within and across Norwegian (L1) and English (L2)?

7 Compensatory theory of L2 reading Elizabeth B. Bernhardt, 2011

8 Methodology: Overlapping constructs Word chains (WC) Find information (FI) Understand main points (UM) Make inferences (MI) Grammar & Vocabulary (GV) Structure & Vocabulary (SV) Aspect of reading L1 L2 Tasks require the reader to decode by separating four words 74 items --- that have been placed as a chain without space between them. Tasks require the reader to locate a piece of explicitly stated 25 items 6 items information one or more places in the text. Tasks typically require the reader to understand the main items points in the text in a holistic fashion. Tasks typically require the reader to reflect and make inferences based on information in the text. Tasks required the reader to choose one of four words or phrases to make the content of one sentence similar to another. The structures being tested were adjective inflection, adjective or adverb, the verb to do in negatives and questions, and indirect speech. Tasks required the reader to move four paragraphs, one at a time, to decide their order. The readers had to look for indicators, like vocabulary, the adverb later, or find out what the pronouns referred to, and how one utterance followed another. 9 items 7 items items items

9 Methodology: Participants & Data collection L1 L2 L1-L2 Invited No reply Refused Accepted Provided data Participated Sample Schools (32%) 42 (12%) 194 (56%) 167 (48%) 244 (70%) 87 (25%) Students (100%) (43%) (58%) (17%)

10 Methodology: Data analysis Step 1 Step 2 Aim Analysis Research question To identify which measures to include in the regression analysis. To identify unique factors in the L2 data. Reliability analysis (L1-L2) Factor analysis (L2) RQ1: To what extent can reading comprehension in English (L2) among first year upper secondary students be explained by reading literacy in Norwegian (L1), and language knowledge in English (L2)? Step 3 To identify explained variance across the L1 and L2 data. Regression analysis I (L1-L2) Step 4 Step 5 To identify characteristics of the 20% lowest performers. To identify explained variance among the 20% lowest performers. Frequency analysis (L1-L2) Regression analysis II (L1-L2) RQ2: What insights about L2 reading literacy can be derived from the 20% lowest performers within and across Norwegian (L1) and English (L2)?

11 Methodology: Reliability analysis Zscores (L2) Number of items Cronbach s Alpha (α) Find information (FI) Understand main points (UM) Make inferences (MI) Grammar, vocabulary (GV) Structure, vocabulary (ST) L2 language knowledge (GR, ST) L2 reading comprehension (FI/UM/MI) ,704 0,683 0,876 0,681 0,878 0,839 0,903 If items deleted 0,649-0,677 0,625-0,666 0,851-0,872 0,613-0,658 0,825-0,862 0,806-0,839 0,893-0,903

12 Methodology: Regression analysis Gender: Knowing the gender of all participants, boys and girls are the only variables. Study programme: This category includes three variables; vocational studies, general studies and unknown. L1 literacy: This category comprises all three L1 sum scores linked to the word chains, the factual text and the fictional text (WC, FI, MI). L2 language knowledge: This category comprises vocabulary, grammar and structure (GV, SV). L2 reading comprehension: This category comprises the three final aspects of reading (FI, UM, MI).

13 Methodology: Categories L1 low L1 high L2 low L2 high The poor readers Low L1 and L2 performance. The 20% lowest performing students in both languages. The low L1- high L2 readers Students in the 20% lowest performing group in L1, while identified in the % groups in L2. The low L2 high L1 readers Students in the 20% lowest performing group in L2, while identified in the % groups in L1. The good readers High L1 and L2 performance. Students in the % quintiles in both languages.

14 Findings I Explained variance Regression analysis based on regression weights

15 Findings I (cont.) 59% explained variance 57% explained by L2 language knowledge (grammar, vocabulary & structure) additional 2% explained by L1 literacy 41% unexplained variance 0,3% gender & study programme

16 Findings II Explained variance Regression analysis based on Bernhardt s theory

17 Findings II (cont.) 59% explained variance 26% explained by L1 literacy additional 33% explained by L2 language knowledge (grammar, vocabulary & structure) 41% unexplained variance 0,3% gender & study programme

18 Discussion Our findings support Bernhardt s compensatory theory However, how do we separate L2 language knowledge from L2 reading comprehension? Does English (L2) reading comprehension consist of component reading skills or one competence? The students read and use language in tandem

19 Findings III Two groups of weak students read differently across L1-L2 The quintiles vary in number from N=1,472 to N=2,401 due to several students achieving the sum score of 26 in L2, which was the cut score between the two top L2 quintiles.

20

21 Findings III (cont.) The poor readers L1 literacy explains 45% L2 language knowledge 0%

22 Boys General Boys General Findings III (cont.) Gender N=364 N= N=505 girls boys boys boys girls girls L1 low & L2 high Sample L2 low & L1 high

23 Findings III (cont.) L1 low L2 high

24 Discussion Why do these students read significantly better in L2 than in L1? Why do boys both outnumber and outperform the girls in this group? Could it be linked to digital assessment, the English language, shorter texts, and topic of interest?

25 Discussion Could the unexplained variance relate to lack of language strategies, that students do not transfer language strategies from one language to another?

26 Conclusion The shift from testing what the students know about language to assessing what students can do with language The challenge L1 literacy and L2 language knowledge are main contributors to the students L2 reading literacy the majority read equally well - or poorly in L1 and L2 a minority read significantly different in L1 and L2 How can test scores be used to make inferences about the needs of student growth - to improve learning and instruction? Is training in transfer of learning strategies one solution?

27 Selected references Bernhardt, E. B. (2011). Understanding advanced second-language reading. NY: Routledge Brevik, L. M. (2013). Research ethics: An investigation into why school leaders agree or refuse to participate in educational research. Problems of Education in the 21st Century, 52, Retrieved from Brevik, L. M. (2014). Making implicit practice explicit: How do upper secondary teachers describe their reading comprehension strategies instruction? International Journal of Educational Research. Doi: /j.ijer Frones, T. S., Narvhus, E. K., & Aasebo, M. C. (2013). Nordic results from the PISA digital reading assessment. Nordic Journal of Digital Literacy, 8(01-02), pp OECD (2009). PISA 2009 Assessment Framework. Key competencies in reading, mathematics and science. Paris: OECD Publications. OECD (2013). PISA 2012 Results: Ready to Learn Students Engagement, Drive and Self-beliefs (Vol. III). Paris: OECD Publishing. doi: / en

28 Thank you, University of Oslo, Norway