The Monitoring of Learning Outcomes in Mali Language of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms

The Monitoring of Learning Outcomes in Mali Language of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms

The Monitoring of Learning Outcomes in Mali Language of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms The William and Flora Hewlett Foundation Grant # 2008-3367 RTI International Project Number: 0212014 Prepared for William and Flora Hewlett Foundation 2121 Sand Hill Road Menlo Park, CA 94025 Prepared by Pierre Varly, on behalf of RTI International 3040 Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 The author s views expressed in this publication do not necessarily reflect the views of the William and Flora Hewlett Foundation or RTI International.

Table of Contents Page List of Figures... v List of Tables... v Brief Overview of National Languages in Mali... 1 Findings from Other Studies on LOI and Teachers Methods and Behaviors (International Association for the Evaluation of Educational Achievement [IEA] Heritage)... 2 Study Objectives... 3 Methodology: Classroom Observations via Snapshot Instruments... 4 Observation Score... 4 Learning Conditions of Pupils in Mali... 5 Do Pupils Really Learn in Their Mother Tongue?... 6 What Are the Languages Used by Teachers Observed in the Classroom?... 8 What Is the Language Used by Pupils?... 8 Individual Versus Group Work... 8 Type of Activities by Teachers and Pupils... 11 Activities and Language Used... 12 Pupils Literacy Environment and Opportunity to Read Index... 13 What are the Effects of Languages Use and Teachers Methods on EGRA Test Scores?.. 14 How Mali Pupils Perform in Reading at Grade 2?... 14 What Inputs Really Have an Effect on EGRA Test Scores?... 15 Proposal of a Framework for Measurement and Action... 18 Conclusions and Recommendations... 19 References... 21 Websites Accessed... 23 ANNEX 1... 24 Regression Models (number of letters), Using Weights... 24 Logit Models (Pupil Could Recognize a Word), Using Weights... 25 Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms iii

ANNEX 2: Correlations Between Test Scores and Classroom Activities... 26 Teachers Activities... 26 Pupils Activities... 26 ANNEX 3: Languages Map in Mali... 27 Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms iv

List of Figures Figure 1. Partial reproduction of the data collection tool... 4 Figure 2. Proportion of language speakers in each classroom... 7 Figure 3. Distribution of teachers work group... 9 Figure 4. Distribution of pupils work group... 9 Figure 5. Relationship between class size and percentage of pupil individual work... 10 Figure 6. Distribution of reading activities by pupils in reading lessons... 12 Figure 7. Distribution of correct word and letters per minute... 14 Figure 8. Characteristics of pupils able to read a word versus others... 15 Figure 9. Share of contribution to letter per minute (clpm) variance by categories of variables in reading lessons... 17 Figure 10. A framework for measurement and action... 18 List of Tables Table 1. LOI and EGRA studies sample... 5 Table 2. Number of lessons by LOI and subjects... 5 Table 3. Learning conditions of pupil... 5 Table 4. Classroom Materials... 6 Table 5. Pupils declaration of language use at home... 6 Table 6. Share of language use by teachers observations by lesson subject... 8 Table 7. Proportion of pupils observations by language used and subject taught... 8 Table 8. Effect of textbook on individual work... 10 Table 9. Teachers activities... 11 Table 10. Pupils activities... 11 Table 11. Share of official LOI use by subject and type of activity... 12 Table 12. Characteristics of pupils able to read a word versus others... 15 Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms v

Brief Overview of National Languages in Mali Among West African Francophone countries, Mali is the most advanced in the introduction of national languages in education. In Burkina Faso, Niger, and Senegal, national languages are introduced as experiments, while in Mali they are the official mode of instruction in many classrooms. Mauritania has also a bilingual French/Arabic system, but Arabic is not the mother tongue of all pupils. National languages were first introduced in Mali in the education system in 1979. In 1987, the Pédagogie convergente method was developed to make a transition between national languages and French, (Traoré 2009). In 2002, national programs were redesigned following a competencies-based approach with the Curriculum bilingue par compétences. Today, in 2,466 curriculum schools, pupils start to learn in 1 of the 11 national languages in early grades before switching to French in grade 4, (MEALN/AFD 2010). Languages diversity, status, categories and data Aménagement linguistique dans le monde from Jacques Leclerc (Université de Laval, Québec) and www.ethnologue.com, maintained by SIL International, are the two main reference sources on language diversity. They are largely used by the United Nations Educational, Scientific, and Cultural Organization (UNESCO), which maintains its own database on endangered languages. The African Academy of Languages (ACALAN) founded by the President of Mali in 2001, and sponsored by the African Union, has a project to map languages and harmonize coding. Africa remains a continent of oral traditions, and the codification of languages is relatively recent. ACALAN (2009) provides few concepts, such as widely spread languages ( dominant language in the literature) or lesser spread languages ( non-dominant language ) and vehicular cross-border languages, Ten vehicular cross-border languages are identified by the academy in West Africa, including Wolof, Pulaar, and Mandingue, and 41 for the whole continent, out of 2000 languages spoken. African languages are split in different linguistic groups: Nilo saharienne, Nigérocongolaise, Bantoue, Khoïsane, and Chamito-sémitique. Language coding is referred as ISO 639-3. In many countries, including Senegal, the national census collects data on ethnic origin rather than on languages used. SIL provides estimated figures on the number of speakers of each language but from different sources and years. It is often not possible to calculate a ratio from a unique source for a given year. However, in the DHS 2006 survey, question 121 asks for ethnic origin, and question 122 asks for language spoken at home ; however, these variables are not used in the report. Moreover, in measuring literacy, only French, Bambara, Malinke, Pulaar, and Songhoi were chosen, and the Arabic script was not used. The percentage of the population literate in their own language cannot be calculated, but it may be possible to calculate the proportion of speakers of each language based on the DHS data According to Jacques Leclerc, Bamanankan is spoken by 2.7 million people as a mother tongue and understood by 4 million people. Indeed, most (2,002 out of 2,466) of the curriculum schools use Bamanankan as the language of instruction (LOI), followed by Songhoi (157 schools) (Agence Francaise du Developpement [AFD] and Ministere de l Education et des Langues Nationales [MEALN] 2010). Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 1

In 2009, the Hewlett Foundation financed a study on reading skills following the EGRA protocol that included a reading test for pupils, questionnaires, and classroom observations (RTI 2009). This study covered 94 schools, where instruments were administered in four national languages (Bamanankan, Bomu, Fulfulde, and Songhoi) to grade 2 pupils. Overall scores were very low, and a great proportion of pupils were not mastering basic competencies expected at grade 2. These low scores are likely a result of one of the three following factors, or a combination of thereof: Living conditions of pupils (e.g., poverty, poor literacy environment, pupils working in the field, and health and nutrition problems); Learning conditions of pupils (e.g., large class size, few textbooks, poor class equipment, temperature); and Inefficient teaching (e.g., low mastering of LOI by teachers, inadequate methods, and insufficient time on task). A recent study on curriculum published on the Ministry of Education (MOE) Web site, (AFD/MEALN 2010) provides some insight on teachers methods and language issues, such as follows: Teachers are not all fully bilingual; LOI can be different from the language spoken home; Reading methods are slow and not motivating for pupils; and Teachers lack of training to teach in national languages. The same statements were made by the former Minister of Education (Traoré 2009). Traditional methods used by teachers, such as repetition and transcription of letters and words written on the blackboard, are not as efficient (RTI 2009). Findings from Other Studies on LOI and Teachers Methods and Behaviors (International Association for the Evaluation of Educational Achievement [IEA] Heritage) IEA s translation procedures allow a comparison of test score results across languages, but IEA studies concentrate on countries where a unique LOI is spoken by the majority of the population. However, IEA has produced extensive work on teachers methods and behaviors, and its studies are a reference framework for measuring and analyzing learning outcomes. The Reading Literacy Study showed that students whose home languages differed from the language of instruction had lower literacy performance students in classes where teachers emphasized reading for comprehension and encouraged more reading also did better. They also cited that availability of books (home, school, nearby community library) was identified as a key factor for high achievements in reading literacy (Postlethwaite 1992). IEA has developed very sophisticated tools for classroom observation via video recording (Stigler 1999) and introduced concepts such as Opportunity to Learn in the Classroom Environment Study. Looking at the difference in teaching in classrooms and teachers behaviors, this study showed that Little time was spent in small group instruction or work.students in some Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 2

classrooms were taught two or three times more of the content than were students in other classroom.home background did not have a direct effect on student achievement (Anderson 1989). IEA work focuses on reading comprehension using paper-pencil tests, while the EGRA protocol is based on further work from neurosciences and early grades using oral fluency measures along with comprehension. Moreover, EGRA targets lowincome and developing countries and has developed its instruments in 74 languages in 41 countries. In developing countries, and specifically in Africa, there is far more linguistic diversity within countries than in developed nations. In addition, literacy environments are poor, particularly in rural areas. At the macro level, the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) grade 6 data show that countries with a high proportion of pupils speaking the test language perform better (Program on the Analysis of Education Systems [PASEC] 2009). However, the relationship is not systematic: Uganda and Kenya have a similar proportion of pupils speaking the SACMEQ test language at home, but their SACMEQ test scores are quite different. PASEC data show that Burundi 1 and Madagascar, where pupils learn in their home language in the early grades, have significantly higher results than other countries in maths. In the Mauritanian bilingual education system, PASEC results are low in French, Arabic, and maths. At the class or individual level, few studies and data 2 address the link between LOI and learning outcomes in Africa. Pupils learning in their mother tongue tend to stay longer in school and repeat fewer grade levels, especially in Mali, but in LOI studies in Kenya and Uganda, Piper (2010) did not find significant impacts of LOI on EGRA test scores. What is the situation in Mali? Study Objectives A large number of studies on Africa, including large scale assessments such as PASEC, show that teachers methods and practices in the classroom and school management are key factors impacting learning outcomes (Varly 2006). Therefore, capturing teachers classrooms practices is a sound base for developing policies aiming at improving learning outcomes. With simple tools, this study will investigate what happens in Mali grade 2 classrooms, using national languages as a medium of instruction. By crossing the classroom snapshot observations with EGRA results, the study will try to identify best practices for improving learning outcomes. Research questions are listed as follows: What are the pupils learning conditions? Do pupils really learn in their mother tongue? What languages are used by teachers and pupils in the classroom? 1 Edstats database accessed September 28, 2010. http://go.worldbank.org/47p3ple940. 2 Pass exam rates used to assess the impact of the use of mother tongue on learning has strong limitations. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 3

How are lessons structured in terms of group or individual work and activities? What are the effects of the type of lessons on learning outcomes compared to contextual affects? How a poor literacy environment and little opportunities to read lead to low reading skills? Methodology: Classroom Observations via Snapshot Instruments Data on teachers methods and pupils activities are collected through snapshot observations. Enumerators in a classroom note the following every two minutes: What are teachers and pupils doing?; and What is the language used? Information on classroom equipment is also collected. Figure 1. Partial reproduction of the data collection tool Snapshot observations Method Activity 1 2 3 4 5 6 Teachers work with a group of pupils Teachers work with an individual pupil Pupils work in a group Pupils work individually Teachers speaks BA BA FR Teachers writes BA Teachers speaks BA BA Teachers writes Pupils speak BA BA BA Pupils read aloud Pupils speak Pupils read silently BA BA BA Note: FR: French language used; BA: Bamanankan language used. This mock sample in Figure 1 shows how the data collection works. Snapshots 1 and 2: Two minutes and four minutes after lesson started, teacher speaks in Bamanankan; pupils speak in group in Bamanankan; and Snapshot 3: Teacher writes in Bamanankan; pupils speak in group in Bamanankan. This data collection form gives us an idea of the lesson structure. Observation Score The data collection took place along with the EGRA protocol on pupils reading competencies. Schools were selected among 16 Centres d Appui Pedagogique (CAPs, administrative districts) in Mali. Some schools or classrooms did not participate in the LOI data collection; a total of 77 classrooms were observed in 60 of the 94 schools covered by the EGRA protocol. The response rate was 63.8% at the school level. See Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 4

Table 1 for an overview of the sample sizes and languages covered by the LOI classroom observation and EGRA data collection. Table 1. LOI and EGRA studies sample LOI data collection EGRA data collection LOI # schools # lessons # schools # pupils Bamanankan 15 22 24 464 Bomu 17 19 25 592 Fulfulde 11 14 20 437 Songhoi 17 22 25 472 Total 60 77 94 1,965 Lesson subjects are classified as reading, maths, and others. Reading lessons account for one third of the collected data. Only two reading lessons were observed in Fulfude and one maths lesson in Songhoi. See Table 2 for an overview of the number of lessons observed. Table 2. Number of lessons by LOI and subjects Class subject LOI Maths Others Reading Bamanankan 5 8 9 Bomu 6 9 4 Fulfulde 3 9 2 Songhoi 1 11 10 Total 15 37 25 Learning Conditions of Pupils in Mali Table 3. The EGRA protocol includes a pupil questionnaire, giving valuable information on the learning conditions of pupils. The data in Table 3 is from the core EGRA study. Learning conditions of pupil % pupils who repeated 5.4% % pupils having a textbook 33.6% % pupils having books at home 17.9% % pupils having homework 51.6% Source: Mali EGRA Report, p.17-18. LOI data provides additional background information at the class level. The average class size is 38.6, 3 but 10% of the classrooms exceed 70 pupils. Only 22.1% of the classrooms had pupils using textbooks. Data also show low attendance rates 3 In Bamako District, the average pupil teacher ratio was 73 in 2008, according to the World Bank (2010). Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 5

(calculated as class size/administrative enrolment). In 25% of the classes, the attendance rate is below 60%, indicating a potential problem in schooling time. See Table 4 for some specific characteristics of classrooms that participated in the data collection. Table 4. Classroom Materials % teachers using a manual or guide in classroom 77.9 % % class with pupils using textbooks 22.1 % % class with writing materials for pupils 85.7 % % classroom with support materials* 64.9 % * Wall art, posters, student writings, other books Source: LOI Dataset. Comments from enumerators indicate further difficulties, such as lack of seats (25% of classrooms), light, or heat. In this harsh environment, pupils are not in good conditions for learning, and the effectiveness of teachers methods, a crucial issue, is also hampered (Abadzi 2006). The introduction of national languages as a medium of instruction can counterbalance the effect of a poor literacy environment and learning conditions. Measuring the value added of using mother tongue as the medium of instruction is the central point of this study, but we will not compare results of pupils using French and using national languages as LOI. Do Pupils Really Learn in Their Mother Tongue? In the EGRA report, 96% of pupils declared learning in their home language. A total of 22.3% of pupils declared speaking more than one language home, and 7.7% reported speaking French, as well. This data was cross-checked during the LOI observation by asking students, in a fun way, to indicate the language spoken at home, with the data measured through a sort of game in the classroom. In the LOI data, 97% of pupils speak their LOI at home. Table 5 shows consistency between pupils individual declaration and LOI data collection. Table 5. Pupils declaration of language use at home Bamanakan Bomu Fuldude Songhoi % pupils declaring speaking LOI (EGRA data) 95.5% 98.1% 92.9% 97.2% % pupils speaking LOI measured in the classroom 98.0% 92.4% 96.0% 98.9% Source: EGRA and LOI datasets. There is very little variation of LOI use at home in the data, but in two classrooms, the proportion of LOI speakers is under 60%. Only 81.9% of classrooms are composed of 100% of LOI speakers. Looking more closely on the composition of the classrooms, pupils may speak several languages at home. If most of the pupils speak the LOI in multilingual homes, we do not know the proportion of LOI use in these households. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 6

Teachers have to face a non-uniform linguistic situation among pupils in their classroom as shown in the Figure 2. Each language is represented as a particular color, with each band representing the proportion of language speakers one of the 94 classrooms of the sample. For instance, in school 91, most pupils speak Bamanankan (blue); in school 1 most pupils speak Songhoi (green). The x-axis represents the percentage of students*, and the y-axis the number of classrooms in the sample. Figure 2. Proportion of language speakers in each classroom * Sum of percentages can be more than 100, as pupils can speak several languages at home. In the EGRA data, Songhoi-speaking pupils are less likely to speak another language. It is worth nothing that Songhoi 4 language does not belong to the same family of languages of Bamanankan, Bomu, and Fulfuldé. This can be also the results of ethnic considerations when marrying or geographical distribution of languages or ethnic groups. See Annex 3 for Mali languages map. 4 http://www.tlfq.ulaval.ca/axl/afrique/mali.htm. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 7

What Are the Languages Used by Teachers Observed in the Classroom? A total of 84.4% of teachers use the official national language as the only medium of instruction; 13% of teachers use French, as well; and 6.5% also use another language. One teacher uses French only (for a math lesson), but no teachers use any French language for a reading lesson. No teachers used more than three languages in the classroom. The language used varies across subject taught, and French is used in maths and others subject lessons (see Table 6). Table 6. Share of language use by teachers observations by lesson subject Official language French Other Reading 99.8 % 0.0 % 0.2 % Maths 91.7 % 8.2 % 0.1 % Others subjects 92.3 % 7.3 % 0.4 % In reading lessons, the designated national language is predominantly used. The classes where French is used are mainly Bomu classes, except one class of Bamanankan. The use of Bomu by teachers and pupils is under 80% on average. What Is the Language Used by Pupils? Similar results are found with pupils language use in the classroom (see Table 7). Pupils speak some French, but this language is not used in reading lessons. Logically, pupils use the same language as teachers, noting, again a variation among subjects taught. Table 7. Proportion of pupils observations by language used and subject taught Official language French Other Reading 100.0 % 0.0 % 0.0 % Math 92.6 % 7.4 % 0.0 % Others subjects 90.3 % 9.3 % 0.4 % Individual Versus Group Work The majority of teachers work with a group of pupils rather than with an individual pupil. No teacher works exclusively with individual pupils, and 5 teachers among 76 work only with the group. The variation of proportion of group work and individual work is important among teachers and pupils as showed in the Figures 3, 4, and 5. In nine classrooms, pupils were only working individually during the observations. Figure 3 illustrates the distribution of time spent by teachers working with the group (the y axis shows percents in each category). As shown in Figure 3, few teachers use Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 8

working with the group as a strategy less than 40% of the time (see low bars on the left in Figure 3). Figure 3. Distribution of teachers work group Repartition of proportion of teachers work group Density 0.5 1 1.5 2 2.5.2.4.6.8 1 Proportion of teachers work group The proportion of group work varies across subjects. Teachers tend to work with a group more frequently in reading lessons, where the proportion is over 50% of observations in all classrooms. In maths lessons, teachers use individual work with pupils more frequently. Figure 4. Distribution of pupils work group Repartition of the proportion of pupils group work Density 0.5 1 1.5 2 0.2.4.6.8 1 Proportion of pupil group work Again, the variations of teachers methods are important within a lesson subject and especially regarding the type of work of pupils are engaged in during maths lessons. There is no typical lesson. Before examining the details of the activities or tasks, we could check if teaching conditions have an impact on methods used. For example, teachers with large classes Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 9

should tend to work more frequently with a group and teachers with fewer LOI speakers would use more individual methods. The availability of textbooks might also explain differences in methods used. Figure 5. Relationship between class size and percentage of pupil individual work Share of pupils individual work and class size Share of pupils individual work 0.2.4.6.8 1 0 20 40 60 80 100 Class size Data show a correlation between pupils individual work and class size. In large classes, pupils work individually less frequently. In classes with more than 60 pupils, the share of individual work is less than 60% of the total observations recorded. Thus, limiting the maximum class size may allow pupils to work individually and may also have an effect on test scores (AIR 2007). We would expect that when pupils use textbooks in the classroom, they would tend to work individually. Teachers use of a book or guide should also have an effect on the lesson structure. Table 8. Effect of textbook on individual work Number Proportion of teachers working with an individual Proportion of pupils working individually Teacher has a book 60 34.6% 52.8 % Ns Teacher has no book 17 38.8% 67.4 % Pupils have textbooks 17 35.3% 58.9 % Ns Pupils have no textbooks 60 36.3% 45.1 % Note: Ns = not significant; * = 10%; ** = 5%; *** = 1%. As shown in Table 8, as expected, when pupils use textbooks (which is only in 22% of the classes), they work individually significantly more. The availability of teachers or pupils book has no effect on the share of group and individual work for teachers. Data show no relation between the proportion of LOI speakers in the classroom and teachers methods. However, while noting that 6.5 % of the teachers used a language * *** Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 10

Table 9. that is not French, nor the official medium, the data does not indicate whether teachers spend more time with pupils who do not speak the LOI at home than with others. When enumerators reported 5 on problematic teachers competencies or language proficiency, the share of group work is significantly higher. When they reported on a good class climate, there is more pupil individual work. Type of Activities by Teachers and Pupils Tables 9 and 10 show that in most cases, teachers main activity is speaking (54.4% of the observations for the reading lessons). According to the subject taught, the type of activity does not significantly change. Teachers activities Activity Reading Maths Others With a group (64.5 %) Teacher speaks 35.9 % 34.6 % 28.6% Teacher writes 13.1 % 17.0 % 13.6 % Teacher reads aloud 13.4 % 8.6 % 10.1 % Teacher does not speak 5.9 % 1.6 % 8.9 % Teacher does not work 0.2 % 0.0% 1.5 % With an individual pupil (35.5 %) Teacher speaks 18.5 % 18.7 % 21.5 % Teacher writes 4.9 % 9.6 % 9.6 % Pupils writing (individually or in group) is the main activity, but there is some variation in the type of pupil work according to subject taught. Table 10. Pupils activities Activity Reading Maths Others In a group (44 %) Pupil speaks 18.5 % 10.1 % 17.3 % Pupil writes 15.1 % 23.1 % 21.5 % Pupil reads aloud 10.1 % 6.1 % 7.5 % Individually (56 %) Pupil speaks 16.9 % 4.8 % 17.5 % Pupil writes 20.3 % 37.1 % 21.3 % Pupil reads aloud 14.5 % 14.9 % 10.6 % Pupil reads silently 4.7 % 4.0 % 4.4 % Writing individually is predominant in math lessons (37.1% of observations). In reading lessons, only 25% of pupils work actually consists in reading (alone or in group, silently or aloud). Silent reading accounts for only 4.7% of total pupils work. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 11

Figure 6 represents the distribution of pupils reading activities (the y axis is the percent of each category). For instance, about 34% of classrooms have no pupils reading activities (see higher bar on left). Figure 6. Distribution of reading activities by pupils in reading lessons Density 0 1 2 3 4 0.2.4.6.8 RATIO_PUPIL_READING There is great variation among classrooms in time spent reading. In one third of the classrooms, pupils do not read themselves in reading lessons; rather, they are listening to the teacher read. A quick comparison of the numbers of observations of teachers and pupils activities show that a non-negligible amount of pupils time is spent just listening, and pupils are therefore not engaged in any activity. Activities and Language Used What are the languages used in the different activities? Most teachers used LOI the major part of the time, especially for the reading lessons. Table 11. Share of official LOI use by subject and type of activity Reading Maths Others Teachers working with a group 100.0 % 91.1% 93.1% Teachers working with individual pupil 99.6 % 92.9% 91.2% Pupils working in a group 100.0 % 91.0% 91.7% Pupils working individually 100.0 % 93.3% 80.0% Although data has limitations, it shows a great variation of teachers practices and allows to us distinguish between teachers using active methods (e.g., involving pupils individual work) and others. Qualitative comments from enumerators also provide useful feedback on what really happens in the classroom. For example, some enumerators reported teachers insufficient mastering of the LOI. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 12

Pupils Literacy Environment and Opportunity to Read Index The data give us a sense of a literacy environment of pupils, but with a limited scope in the context of the matrix designed by Shiohata (2010). Data do not give information on the literacy environment outside home and school, which is still where kids are more likely to learn to read. Pupils could have opportunities to read themselves in and outside of school. In our data, only 17.9% of pupils have books or newspapers at home, and the proportion varies between regions. Assuming Bamako and Tombouctou districts are urban areas, there are large differences in the data with regards to access to reading resources at home between rural and urban pupils as reported at the household level in the DHS (MACRO 2007). DHS data showed that 1.4% of females and 5% of males read a newspaper at least once a week in rural areas versus 13% of females and 22.8% of males in urban area. In rural areas, pupils are not likely to have access to any reading support home but the key factor is logically parents literacy. When men or women have at least secondary education, they have over a 40% chance to read the newspaper at least once a week but less than 8% when they have only primary education. Only 0.1% of women having no formal education read a newspaper at least once a week (MACRO 2007). Opportunity to Read (OTR) Index We can elaborate an OTR Index based on Gillies (2008) proposal, SHIOHATA (2010), as well as pioneer IEA work. The idea is to capture the likelihood of pupils to read at school and at home altogether. Several variables can be introduced in the OTR Index. One person can read at home; Pupil has reading textbook, and pupil has other book home; Pupil has homework; Time devoted to reading activities in the classroom (estimated at the class level); Attendance rate (at the class level); Pupil has been absent; and Posters or wall decorations or exposure to writing in the classroom. Pre-schooling was not included in the indicator, as it is not sure whether children are prepared or learned to read at this stage in the various forms of pre-schooling. Moreover, IEA separates Early Home Literacy and Home Educational Resources in PIRLS study.1. The index is a Z-score (mean 0 and standard 1), little correlated with the share of reading activities component (correlated [cor.] 0.06). In Senegal, teaching hours per week and share of reading activities in the classroom combined allowed us to estimate a weekly effective time on task reading, strongly correlated with the OTR Index (cor. 0.74). Here our index is correlated with SES (cor. 0.2), but very little correlations were found with letters and words per minute. The weekly hours of reading declared by the director seems to be a much better predictor of test scores than the simple share of effective time on task reading measured in a lesson. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 13

This OTR Index allows comparison of effects size of the literacy environment versus teachers methods on EGRA results. What are the Effects of Languages Use and Teachers Methods on EGRA Test Scores? In order to improve pupils learning outcomes and reading skills measured through the EGRA protocol, the key issue is to identify what type of teachers methods are the most efficient given the learning context and the pupils background characteristics. After looking at the EGRA results, according to different variables, regression models (see Annex 1) will be used to examine whether teachers methods and pupils activities have a real impact on EGRA test scores. Unfortunately, there is no information on the characteristics of teachers and school heads, but data from pupils questionnaires and classroom observations provide us with sufficient information to produce control variables and estimate net effects. How Mali Pupils Perform in Reading at Grade 2? It is important to note that tests were administered in four different languages and no equating procedure was used. Mali EGRA test scores are very low. According to the EGRA report, at the end of grade 2, between 10% and 33% of pupils were not able to indicate the beginning and the end of a written sentence or the right direction of reading. A total of 26% of Bamanankan, 37% of Bomu, 22% of Fulfude and 49% of Songhoi speakers were not able to read any letter. The average number of words correctly read in one minute varies between one and two, (RTI/CEPROCIDE 2009). The majority of pupils cannot read a word or recognize a grapheme as shown in the distribution in Figure 7. Figure 7. Distribution of correct word and letters per minute Density 0.05.1.15 0 20 40 60 80 Correct Letters Per Minute Density 0.1.2.3.4.5 0 10 20 30 40 50 Correct Words Per Minute In this situation, for the words per minute, we will split the pupils population in two groups: (1) pupils than can manage to do something; and (2) those who cannot. There are too few observations to make reasonable comparisons between pupils that acquired basic skills (reading more than 50 words per minute for instance) and those who did not. We will use letter and word recognition as key indicators of reading skills since too few pupils are able to correctly understand what they read. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 14

The response variables to be considered in the following section are: Number of letters read; and Pupil could read at least a word. What Inputs Really Have an Effect on EGRA Test Scores? In a first attempt to measure the effects of LOI and other variables on pupils success in reading a letter/word, we will compare the two sub-populations. SES is also calculated as a Z-score. Figure 8 shows that pupils background characteristics, such as socioeconomic status and OTR, seem to generate a significant difference in pupils success in reading a word. Results in Figure 8 are presented for all types of lessons. Figure 8. Characteristics of pupils able to read a word versus others However, when looking at the large confidence intervals ( conf. ) in Table 12, we cannot conclude on any effects on these variables on the likelihood of success. Table 12. Characteristics of pupils able to read a word versus others Pupils failed Pupils can read a word Mean Conf. Mean Conf. OTR 0.10-0.23 0.42 0.38 0.00 0.76 SES 0.41 0.15 0.66 0.51-0.06 1.07 Teacher share of group work 0.73 0.65 0.82 0.76 0.70 0.81 Pupil share of group work 0.54 0.46 0.63 0.48 0.39 0.56 Teacher use of LOI 0.99.98 1.00 1,00 0.99 1.00 Number of pupils 1,055 140 Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 15

Using regression and logit models It is necessary to consider living and learning conditions and teachers methods all together in regression and logit models in order to check for effects. As relationships between teaching environment (such as class size) and teachers methods were identified, environment and teachers methods could have joint effect on test scores. For instance, the use of book or guide by teacher has large size effects on the grapheme and word recognition competencies, but also on teachers methods. Similarly, we already identified a correlation between socioeconomic status and OTR Index. Logit and regression models are suitable to control for living and learning environment effects. In order to test the robustness of the different models, several response variables and specifications are used: number of letters or words read in one minute and binary success variables (pupil can read a word or a letter). Weighted and nonweighted estimation results will be also cross checked. Only logit models using weights will be presented below In each school, one class was randomly selected to take the EGRA test but several lessons for the same class could be observed in the LOI data collection. In order to merge EGRA and LOI data, one lesson was selected in each school with multiple observations: the reading lesson, when possible, or the other subject lesson, resulting in 59 schools and 1,197 pupils. When merging LOI and EGRA datasets, we noticed that schools that took the LOI protocol had significantly lower letters recognition test scores than others, indicating a potential selection bias. This was not treated in the analysis. The repartition of pupils per lessons subject is as follow maths (158), others, (544) and reading (495). The models presented below will be for reading lessons versus maths and others. Imputation was made on missing data, using the most frequent modality. Variables whose response rates were under 80 % were not introduced in the models. In the final models, the number of observation is 988.. Figure 9 shows how the different categories of variables contribute to the variance of letters read per minute, in reading lessons. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 16

Figure 9. Share of contribution to letter per minute (clpm) variance by categories of variables in reading lessons We found similar patterns in Senegal (RTI 2011), but the variance of the response variable (letters per minute) is much smaller in Mali (characteristics and prior schooling contributes to the major part of the variance explanation (9.05%). The OTR index contribution is 2.84%, but does not include the hours of reading per week cited by teachers as in Senegal. Teachers methods and behaviors contribute to 2.25% of variance. Teachers behaviors include qualitative comments from the enumerators on teachers competencies (e.g., LOI proficiency, lessons structure) and class climate. These qualitative comments contribute far more than teachers methods measured in the LOI data collection. It is an indication that qualitative observations can be a better way to capture how teachers work rather than the snapshot data collection. Class effect is calculated by including dummy variables for each school and gives a sense of the differences generated when a pupil is enrolled in school A versus a school B (Varly 2006). A total of 75.5% of the variance remains unexplained versus 67.9% in Senegal, using roughly same procedures (but without weights). Presenting variance decomposition using logit models is not as straightforward as with regression models using quantitative response variable. The OTR index was disaggregated in order to check for effects of each component item. Only a few variables are significant across the different models tested (see Annex 1). Considering use of LOI by teachers, no effect was found in the reading lessons, but there is very little variation. The share of reading activities by pupils only has an effect on word recognition. More pupil individual work in maths and other lessons is associated with a greater chance to read at least a word, but the effect is not stable across the models. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 17

Pupils individual writing is negatively correlated with the test scores (see Annex 2) and might be simply recopying what is on the blackboard. Reading Aloud Individual is associated with better letter recognition, but it should be duly noted that this last activity is also the methodology of the EGRA testing; in other words, pupils frequently reading aloud individually may more prepared for the test for that reason. No teachers specific activity is correlated with the pupils results, see Annex 2. Females have significantly lower scores in letter recognition, which is also impacted by socioeconomic status and Opportunity to Learn. Teacher using a book variable have relatively large effects in all the models tested. The ratio of writing supports is associated with better letter recognition. When enumerators reported on a good class climate and good teachers competencies, the pupils results were better. Repetition has a negative impact on word recognition, but the repetition rate measured in our data is quite low in grade 2 in curriculum classrooms (6%). The results of the pupils seem to be determined by basic inputs (including teachers language proficiency and ability to follow instructions on how to sequence a lesson), rather than by teachers methods measured through the share of group/individual work. Snapshot observations only capture a sample of what is happening during a lesson. Proposal of a Framework for Measurement and Action From existing literature (Abadzi 2006, Verspoor 1991, Pasec 2009, Gillies 2008, Shiohata 2010) and looking at recent EGRA reports and findings (Gove 2010), as well as commonly reported problems affecting use of national languages as LOI as, a framework can be produced to categorize factors influencing specifically reading skills and identify areas of action (see Figure 10). Figure 10. A framework for measurement and action Categories Individual characteristics Items Age, gender, rank in the sibling. Measurement methods Questionnaire Prior schooling Nursery, repetition, disruption in schooling Questionnaire Living conditions Water supply, number of goods possessed, nutrition, sibling size. Questionnaire Pupils Ethno-linguistic background Literacy environment* Ethnic origin, languages spoken home and outside home Parents literacy, books home, classroom posters. Questionnaire Questionnaire Learning conditions* Textbook, class size, home work, Questionnaire and observations Parents attitudes towards national language Do parents want their children to learn in the home language? None Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 18

Categories Items Measurement methods Teachers method Group vs. individual work, activities, time on task, Observations Teachers and Schools Teachers and school characteristics Teachers language proficiency Use of languages in the classroom Status, age, gender, initial training, public school LOI and national languages Time spent using languages Questionnaire None Observations Teachers mental representations Pedagogical objectives, best perceived methods, LOI Questionnaire Conclusions and Recommendations The teaching conditions in Mali are not conducive to learning to read. The students are in overcrowded classrooms and have no textbooks. Very few have access to books or newspapers at home and absenteeism is widespread. Teachers do not have enough materials for teaching reading and do not give sufficient attention to individual reading methods. The structure of lessons varies widely among teachers. The test administrators have expressed concerns about teachers mastering the LOI and the difficulties teachers encounter in implementing teaching methods promoted by educational authorities. Thus, few teachers are able to teach their students how to read, whether they mostly use the mother tongue of students or not. Given the low variation of language use, no effect could be demonstrated in this study. However, more individual reading by pupils seems to have positive effects on acquiring reading skills. Priority measures should aim at ensuring that pupils are in the minimum conditions of comfort for learning. The number of students should be limited to 60 in all classrooms. Textbooks should be widely distributed. In the actual configuration, the effective reading time does not match minimum standards, and teachers methods have only marginal effects. Teachers' skills should be systematically assessed regarding the mastery of basic language skills and in applying appropriate teaching methods out of the most common repetition and transcription. Indeed, the use of LOI does not have the expected effects on test scores. National languages and French test results are very similar; looking at curriculum classrooms only, use of LOI does not have a significant impact on reading skills. Several administrators reported on teachers having problems of language proficiency. Strong measures must be taken to change the practices of teachers in classrooms, either through the inspectorate, teachers unions, or provision of capacity building through initial or in-service training. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 19

A new kind of social deal between teachers and the public should be considered. If the quality of teaching is not improved, this could jeopardize the positive trends in school enrollment rates recorded in coming years. While the introduction of national languages has reduced repetition and dropout rates, it did not, alone, significantly improve the quality of learning. Government policy should focus on reducing the gap between official statements and reality of Malian classes in terms of teaching methods and provide decent conditions of learning and teaching to all. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 20

References Abadzi (2006), Efficient learning for the poor: insights from the frontier of cognitive neuroscience. World Bank, Washington. ADEA (2005), Optimiser l apprentissage et l éducation en Afrique le facteur langue. Conférence sur l éducation bilingue et l utilisation des langues locales. UNESCO. Anderson L.W, et al. (1989). The IEA Classroom Environmental Study. Oxford: Pergamon Press. AIR (2007). Large class size in the developing word: what we know and what can we do?. USAID/EQUIP1. AFD/MEALN (2010), Etude sur le curriculum de l enseignement fondamental, Développement du scénario privilégié. Banque Mondiale (2010). Le système éducatif malien, Analyse sectorielle pour une amélioration de la qualité et de l efficacité du système, Document de travail de la Banque Mondiale no 198, RESEN, Banque Mondiale, Washington. http://www.education.gov.ml/img/pdf/wp198_mali10_text_1_.pdf. Gillies J., QUIJADA J. (2008). Opportunity to Learn: A high impact strategy for improving educational outcomes in developing countries. USAID/EQUIP2. Gove A. & Cvelich P. (2010). Early Reading: Igniting Education for All. A report by the Early Grade Learning Community of Practice. Research Triangle Park, NC: Research Triangle Institute. Hewlett Foundation (2008), The Quality Education in Developing Countries Initiative Grantmaking Strategy. Lookheed M., Verspoor A. (1991). Improving Primary Education in Developing Countries. Oxford, Oxford University Press. MACRO International (2007), Mali Demographic and Health Survey 2006. MACRO International Inc. MBODJI-POUYE A., VAN DEN AVENNE C. (2009), Code switching in rural Mali, pp.301 312, Bristol Papers in Education n o 4. Symposium Books. PASEC (2009). Les apprentissage scolaires au Burkina Faso: les effets du contexte, les facteurs pour agir, Evaluation diagnostique Burkina Faso. PASEC/CONFEMEN, Dakar. PIPER B (2010). Kenya Early Grade Reading Assessment Findings Report. RTI International. PIPER. (2010). Uganda Early Grade Reading Assessment Findings Report. RTI International. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 21

Postlethwaite T.N., Ross, K.N. (1992). Effective Schools in Reading: Implications for Educational Planners: An Exploratory Study, The IEA Study of Reading Literacy II. RTI/CEPROCIDE (2009). Evaluation des Compétences Fondamentales en Lecture des Elèves de 2ème Année des Ecoles Bamanankan, Bomu, Fulfuldé Et Songhoï du Premier Cycle de L Enseignement Fondamental. RTI International. RTI (2011). The Monitoring of Learning Outcomes in Sub-Saharan Africa: Senegal. Pupils. RTI International. SHIOHATA M. (2010). Exploring the literacy environment: a case study from urban Senegal. Comparative Education Review, vol. 54, no.2. Stigler J.W. & al (1999). The TIMSS Videotape Classroom Study: Methods and Findings From an Exploratory Research Project on Eighth-Grade Mathematics Instruction in Germany, Japan, and the United States. (NCES 1999-074). U.S. Department of Education, Washington, DC: National Center for Education Statistics. TRAORE M. L. (2009). L utilisation des langues nationales dans le système éducatif malien : historique, défis., Perspectives in Languages and Education in Africa, pp. 155 161. Bristol Papers in Education n o 4, Symposium Books. UNESCO (2008). Mothe tongue matters: local language as a key to effective learning, Mali Case Studies, p. 9 16. UNESCO. UNESCO (2010). Why and how Africa should invest in African languages and multilingual education: An evidence- and practice-based policy advocacy brief. UNESCO Institute for Lifelong Learning. Varly P. (2006). Gestion scolaire et réussite éducative, Exposé de cadrage des Assises Francophones sur la gestion scolaire. Document de travail, CONFEMEN. VARLY P. (2010). L apprentissage de la lecture au Mali. Un blog sur l'éducation dans les pays du Sud. Retrieved from http://varlyproject.wordpress.com/. VARLY P. (2010). Rapport de mission pour RTI EGRA-Sénégal & Mali, Appui au séminaire de communication et mobilisation sociale. Document de travail. VARLY P. (2010). Language of instruction and teachers methods in Senegal grade 3 classrooms, Draft paper. RTI International. Wechtler, et al. (2007). "The cost-effectiveness of inputs in primary education: Insights from recent student surveys for sub-saharan Africa," Proceedings of the German Development Economics Conference, Gättingen 2007 5. Verein für Socialpolitik, Research Committee Development Economics. Mali: Languages of Instruction and Teachers Methods in Mali Grade 2 Curriculum Classrooms 22