Is the PISA test more female friendly than PIAAC measurement of skills? New evidence on gender differences

Transcription

1 Is the PISA test more female friendly than PIAAC measurement of skills? New evidence on gender differences Kjersti Lundetræ & Oddny Judith Solheim readingcentre.no

2

3 Gender differences in reading Greater gender differences in favour of girls among 15-yearolds in the Nordic countries than the international average (except Denmark). Gender differences in reading seem to increase throughout schooling and then decrease or even disappear with age. In PIAAC, statistically significant gender differences in reading literacy were found in fewer than half of the 33 countries surveyed mostly in favour of men. No significant gender differences in reading in the youngest age group (16 24-year-olds), in the Nordic countries. 3

4 What can explain gender differences in reading? Cognitive differences? Feminisation of school? The teaching methods used? Gender differences in reading frequency? Gender differences in reading motivation? Assessment features? 4

5 Assessment features and gender differences Greater gender differences in favour of girls for narrative texts than for informational texts. Greater gender differences in favour of girls for continuous texts than for non-continuous texts. Greater gender differences in favour of girls on constructedresponse items than on multiple-choice items. Larger gender gaps for reflecting on text than for interpreting text and retrieving information in both PISA 2000 and

6 Research questions 1. What is the magnitude of the effect of gender on literacy across PIRLS (10-year-olds), PISA (15-year-olds) and PIAAC (16 24-year-olds) in Denmark, Finland, Norway and Sweden? 2. Are there any differences in the operationalisation of reading literacy, test characteristics and implementation between PIRLS, PISA and PIAAC with regard to aspects that previous research has found to have an impact on gender differences in reading? 6

7 Sample sizes of PIRLS 2011, PISA 2009 and PIAAC 2012 PIRLS 2011 PISA 2009 PIAAC 2012 (10-year-olds) (15-year-olds) (16 24-year-olds) Denmark 4,594 5,924 1,064 Finland 4,640 5, Norway 3,190 4, Sweden 4,622 4, Total 17,046 20,961 3,765 7

8 Definitions of reading literacy in PIRLS, PISA and PIAAC PIRLS Reading literacy is defined as the ability to understand and use those written language forms required by the society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment. PISA Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one s goals, develop one s knowledge and potential, and participate in society. PIAAC Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one s goals, and to develop one s knowledge and potential. 8

9 Definitions of reading literacy in PIRLS, PISA and PIAAC PIRLS Reading literacy is defined as the ability to understand and use those written language forms required by the society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment. PISA Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one s goals, develop one s knowledge and potential, and participate in society. PIAAC Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one s goals, and to develop one s knowledge and potential. 9

10 Definitions of reading literacy in PIRLS, PISA and PIAAC PIRLS Reading literacy is defined as the ability to understand and use those written language forms required by the society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment. PISA Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one s goals, develop one s knowledge and potential, and participate in society. PIAAC Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one s goals, and to develop one s knowledge and potential. 10

11 Definitions of reading literacy in PIRLS, PISA and PIAAC PIRLS Reading literacy is defined as the ability to understand and use those written language forms required by the society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment. PISA Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one s goals, develop one s knowledge and potential, and participate in society. PIAAC Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one s goals, and to develop one s knowledge and potential. 11

12 Definitions of reading literacy in PIRLS, PISA and PIAAC PIRLS Reading literacy is defined as the ability to understand and use those written language forms required by the society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment. PISA Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one s goals, develop one s knowledge and potential, and participate in society. PIAAC Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one s goals, and to develop one s knowledge and potential. 12

13 Cohen's d Gender differences in reading literacy in favour of the girls in the Nordic countries PIRLS 2011 PISA 2009 PIAAC 2012* Denmark Finland Norway Sweden *16-24-year-olds 13

14 Text types used in PIRLS, PISA and PIAAC (percentages of all texts) PIRLS 2011 PISA 2009 PIAAC 2012 Fiction ( Narrative in PISA) Non-fiction Continuous Discontinuous Mixed + multiple

15 Item formats in PIRLS, PISA and PIAAC PIRLS 2011 PISA 2009 PIAAC 2012 Constructed response a Selected response 100 b Multiple choice a Pencil-and-paper version. b Computer-based version. 15

16 Aspects of reading measured in PIRLS, PISA and PIAAC (percentages of all items) PIRLS 2011 PISA 2009 PIAAC 2012 Access and retrieve Infer and interpret Evaluate and reflect

17 Implementation in PIRLS, PISA and PIAAC Sampling procedures PIRLS class PISA group within school PIAAC individual Participation PIRLS and PISA had to if parents gave consent PIAAC voluntary Time spend PIRLS: 2 x 40 min min. questionnaire PISA: 2 hours + 30 min. questionnaire PIAAC: 50 minutes + 40 min. questionnaire Incentive PIRLS PISA PIAAC offered in some form in the Nordic countries 17

18 Conclusions A reading survey should give girls and boys the same starting point for realising their potential as readers! The surveys differ on points that previous research has found to affect the magnitude of gender differences in reading. PISA is more girl-friendly than PIAAC. PIAAC is more boy-friendly than PISA. Differences in assessment features between the international reading surveys have implications for validity. Is the operationalisation of reading literacy in PIRLS and PISA more valid than that used in PIAAC, or the other way around? 18

19 For future reading assessments 1. Consider that the extent to which students reveal their full potential as readers in an assessment situation is affected by features of the assessment, and possibly also by the implementation of the assessment. 2. Include an assessment task reporting large gender differences in PISA in a PIAAC assessment, and conversely include an assessment task reporting no gender differences in PIAAC in a PISA assessment. 19

20 References Kirsch, I., de Jong, J., Lafontaine, D., McQueen, J., Menedelovits, J., & Monseur, C. (2002). Reading for change: Performance and engagement cross countires. Results from PISA Paris: Organization for Economic cooperation and Development. Lafontaine, D., & Monseur, C. (2009). Gender gap in comparative studies of reading comprehension: To what extent do the test characteristics make a difference? European Educational Research Journal, 8, Mullis, I. V. S., Martin, M. O., Gonzalez, E. J., & Kennedy, A. M. (2003). PIRLS 2001 international report: IEA s study of reading literacy achievement in primary schools in 35 countries. Chestnut Hill, MA: International Study Center, Lynch School of Education, Boston College. Mullis, I. V. S., Martin, M. O., Kennedy, A. M., & Foy, P. (2007). PIRLS 2006 international report: IEA s progress in international Reading Literacy Study in primary schools in 40 countries. Chestnut Hill, MA: International Study Center, Lynch School of Education, Boston College. Mullis, I. V. S., Martin, M. O., Foy, P., & Drucker, K. T. (2012). PIRLS 2011 international results in reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. OECD (2001). Knowledge and skills for life: First results from PISA Paris: OECD Publishing. OECD (2010a). PISA 2009 results: Learning to learn Student engagement, strategies and practices (Vol. III). Paris: OECD Publishing. doi: / en OECD (2010b). PISA 2009 results: What students know and can do Student performance in reading, mathematics and science (Vol. I). Paris: OECD Publishing. doi: / en OECD (2013a). OECD skills outlook 2013: First Results from the survey of adult skills. Paris: OECD Publishing. Roe, A., & Taube, K. (2003). Reading achievement and gender differences. In S. Lie, P. Linnakylä, & A. Roe (Eds.), Northern lights on PISA (pp ). Oslo: Department of Teacher Education and School Development, University of Oslo. Routitsky, A., & Turner, R. (2003). Item format types and their influence on cross-national comparisons of student performance. Paper presented at the American Educational Research Association (AERA), Chicago. Retrieved from Schwabe, F., McElvany, N., & Trendtel, M. (2015). The school age gender gap in reading achievement: Examining the influences of item format and intrinsic reading motivation. Reading Research Quarterly, 50, Statistics Canada & OECD (2005). Learning a living: First results of the adult literacy and life skills survey. Ottawa: OECD. 20