Unlocking Talent: Evaluation of a tablet- based Masamu intervention in a Malawian Primary School

Unlocking Talent: Evaluation of a tablet- based Masamu intervention in a Malawian Primary School DR NICOLA PITCHFORD Associate Professor University of Nottingham

SUMMARY This study evaluated the tablet- based Masamu intervention, developed by EuroTalk, in supporting the acquisition of mathematical abilities in primary school children in Malawi. A Randomised Control Trial (RCT) was conducted in a medium- sized primary school in Lilongwe in which 400 children from Standard 1-4 were randomised to one of three intervention groups: a tablet- based Masamu intervention group, a tablet- based Non- Masamu control group, and a normal- practice control group. Children were pre- tested using tablets at the start of the school year on two tests of mathematical knowledge and a range of basic skills related to scholastic progression. Ambitions and attitudes towards learning were also measured. The intervention was then delivered over an 8- week period, for the equivalent of 30 minutes per day, by classroom teachers at the school, with technical support from VSO. Children were then post- tested on the same assessments as given at pre- test. A matched sample of 283 children, from Standard 1-3, was analysed to investigate the effectiveness of the tablet- based Masamu intervention. Results showed significant effects of the tablet- based Masamu intervention over and above normal classroom practice or using tablets without the Masamu software. In general, children that had received the tablet- based Masamu intervention made signifcantly greater gains in mathematical ability over the 8- week intervention period than both groups of controls. Furthermore, the greater learning gains shown by the Masamu intervention group compared to the control groups on the tablet assessments transferred to paper and pencil format, illustrating that the mathematical knowledge acquired through using tablets generalises to different contexts. In conclusion, this study shows that the tablet- based Masamu intervention developed by EuroTalk is more effective than current pedagogical practice in supporting the development of mathematical knowledge in primary school children in Malawi. Future considerations for maximising effectiveness are given. February 02, 2014 2

TABLE OF CONTENTS 1.0 Background 5 2.0 Aims of study 7 3.0 Methodology 8 3.1 Design 8 Figure 1. Headteacher, Deputy Head Teacher, and Regional Primary School Advisor for Biwi Primary School Lilongwe 8 3.2 Participants 8 Table 1. Composition of the study sample at each stage of the RCT 10 Table 2. Descriptive statistics of the matched sample 11 3.3 Interventions 12 Location of Intervention Figure 2. Images of the Unlocking Talent Learning Centre built at Biwi Primary School, Lilongwe 14 3.4 Assessments 14 Basic Skills 15 Mathematical Ability 15 Mathematical Concepts (MC) Maths Curriculum Knowledge (CK) Maths Curriculum Knowledge Generalisation (CKG) Ambitions and Attitudes 16 Figure 3. Illustrations of the tasks included in the assessment app 17 3.5 Procedure 18 Enrolment 18 Figure 4. Researcher and assistant enrolling a child for the study 19 Pre- test and post- test assessments 19 Pre- test and post- test assessments 19 Figure 5. Images of the pre- test and post- test assessments 20 Interventions 20 3.6 Data analysis 21 4.0 Results 23 1) Effectiveness of the EuroTalk Masamu Tablet Intervention 23 Table 3. Intervention group performance at pre- test and post- test 24 2) Most appropriate Standard in which to implement the EuroTalk 25 Masamu Tablet Intervention Table 4. Effect size (Cohen s d) analyses 25 3) Difference across girls and boys to the EuroTalk Masamu 25 Tablet Intervention Figure 6. Gender histogram 26 4) Basic skills related to mathematical ability 26 Reliability and valididty of the new assessment app 27 Table 5. Reliability of the new assessment app 28 Table 6. Validity of the new assessment app 29 3

5) Basic skills profile of low achievers in mathematical ability 29 Standard 2 29 Standard 3 30 6) Changes in ambitions and attitudes towards learning 30 Table 7. Pre and post intervention performance on the AAQ 31 5.0 Conclusions 32 Figure 7. Developmental trajectory of maths curriculum knowledge 32 6.0 Future considerations 34 7.0 References 35 8.0 Acknowledgements 36 9.0 Appendices 37 Appendix I: Standard 1 ANOVA summary tables 37 Appendix II: Standard 2 ANOVA summary tables 38 Appendix III: Standard 3 ANOVA summary tables 40 Appendix IV: Gender ANOVA summary table 42 Appendix V: Partial correlations between basic skills 43 and mathematical abilities Appendix VI: Pre and post intervention performance on the 44 Ambitions and Attitudes Questionnaire (AAQ) for the matched sample of children 4

1.0 BACKGROUND Research with digital educational software has shown increased motivation (Rosas et al., 2003) and promotion of positive attitudes (Ke, 2008) towards mathematics in primary school children. However, a recent study has concluded that although technology is used in many classrooms in the West, its potential to support learning is often underutilized due to limitations in its design and content (Yelland & Kilderry, 2010). Consequently, findings regarding the attainment benefits of technology- based interventions are currently limited and contradictory (Sanford et al., 2006). For example, a large- scale study by the US Department of Education that compared three technology- based maths interventions in 6 th Grade pupils across the US reported no significant improvement in test scores over a school year compared to pupils that had received normal pedagogical practice. Furthermore, differences in test scores were not related to any of the school and classroom characteristics measured (Dynarski et al., 2007). Yet children enter primary school with a natural affinity for mathematics. Studies have shown that even babies can discriminate sets of objects that vary in number (Lipton & Spelke, 2003). Also, when children begin primary school they come with an enthusiasm for learning, which if captured and supported, could propel them towards reaching their full potential. Thus, well- designed technologies that optimise learning could serve to support development of mathematical ability in the early primary years, and provide a solid foundation on which to build further, more complex, abilities. Promoting understanding of mathematics in the early primary years is critical, as longitudinal research has shown that early mathematical understanding is highly influential on later mathematics and reading attainment at school (Duncan et al., 2007), even after controlling for other basic skills that are known to impact on scholastic attainment (Siegler et al., 2012). According to the UNESCO- IBE World Data on Education Report (2010) (http://www.ibe.unesco.org) standards in mathematics across Malawi are very poor. In tests conducted in 2004, 98% of pupils in Malawi did not possess mathematical skills beyond basic numeracy and Malawi was ranked the second lowest for mathematics of the 14 participating countries in the report. Similar findings were reported by USAID in tests conducted across 50 primary schools in Malawi in 2010 (http://pdf.usaid.gov/pdf_docs/pnaec139.pdf). The USAID report stated that children could only answer the most elementary and procedural of items with any sense of confidence (p. 42). Thus radical shifts in pedagogy are needed in Malawi to raise academic standards. Recent advances in digital technology could provide an alternative means of pedagogical support to current classroom practices if technology- based interventions are shown to be effective. Mobile technologies, such as tablets, might be particularly suited to developing countries like Malawi, where class sizes are typically large (average 76 primary school children per class in Malawi in 2011 according to the UNESCO Institute for Statistics, http://stats.uis.unesco.org) and classroom numbers are usually small, leading to severe overcrowding. In addition, the quality of teaching in Malawi can vary greatly and many teachers may have 5

received poor training. Coupled with few resources to assist with teaching, the learning environment is often extremely impoverished. Consequently, within the current primary school system in Malawi it is difficult to track developmental progression of individual children, identify those that are falling behind, and provide child- centred individual tuition at a consistent level of quality. Digital technologies, such as tablets, afford many advantages to both the classteacher and their pupils, if the software is well designed and the content is grounded in a solid well- constructed curriculum that is appropriate for the child s developmental stage. Even in crowded or outside classrooms, mobile technologies, such as tablets, can deliver one- to- one interactive instruction, with clear objectives, in a consistent manner to all children, thus equating teaching quality across pupils. Children can repeat material as often as they need, thus the pace of learning is tailored to individual needs. Individual progress can also be monitored objectively and easily, using assessments built into the software. With overcrowded classes and limited resources in Malawi, tablet- based interventions could help to radically improve academic standards across the primary years. This study reports a scientific investigation into the effectiveness of a tablet- based intervention to support mathematical ability in primary school children in Malawi. EuroTalk developed the tablet- based Masamu intervention that was evaluated. Based on the National Primary Curriculum for mathematics delivered throughout primary schools in Malawi it is given in Chichewa - the official language in Malawi - via individual electronic tablets (ipad minis) and incorporates all of the features of well designed software outlined above. With the support of the Scottish Government, EuroTalk have trialled the intervention in 30 schools to date. They have commissioned this full scientific evaluation of the effectiveness of the intervention, prior to considering further roll- out. 6

2.0 AIMS OF STUDY Principal aims: 1) Evaluate the effectiveness of the EuroTalk tablet- based Masamu intervention over normal classroom practice in supporting the development of mathematical ability in primary school children in Malawi. 2) Establish the most appropriate Standard in which to implement the EuroTalk tablet- based Masamu intervention. 3) Determine if girls respond differently to the EuroTalk tablet- based Masamu intervention than boys. Secondary aims: 4) Ascertain the basic skills related to mathematical ability. 5) Identify low achievers in mathematical ability and the associated basic skills underpinning their abilities. 6) Investigate if the EuroTalk tablet- based Masamu intervention alters children s ambitions and attitudes towards learning compared to normal classroom practice. 7

3.0 METHODOLOGY 3.1 Design A Randomised Control Trial (RTC) was conducted in which children from Standard 1-4 at a medium- sized primary school (Biwi Primary School) situated in an urban area of Lilongwe (capital of Malawi) were randomised to one of three groups: RED group = Masamu Tablet Intervention BLUE group = Non- Masamu Tablet Intervention PINK group = Normal Practice Children were tested on mathematical ability and basic skills associated with scholastic progression immediately before (pre- test) and after (post- test) the intervention period. The intervention period lasted for 8 weeks. Figure 1. Headteacher, Deputy Head Teacher, and Regional Primary School Advisor for Biwi Primary School Lilongwe. 3.2 Participants Table 1 summarises the composition of the study sample at each stage of the RCT. In total, 436 children were enrolled to the study. These were all children from Standard 1-4 attending Biwi Primary School on the first two days of the school year (2013-14). Due to constraints on intervention group size, 36 children were excluded from the study. The remaining 400 children were randomised to one of three intervention groups. Of these, 385 children were pre- tested, using a tablet- based assessment app developed specifically for this study. Children that were randomised to intervention group but not pre- tested (15 children in total) were either absent (7 children) or had transferred school (8 children) when the pre- testing took place. 8

Standard 4 children did not receive the intervention and served as normal classroom practice controls to establish the developmental trajectory of mathematical ability without tablet- based Masamu intervention. Accordingly, 115 children from Standard 1-3 received the tablet- based Masamu intervention (RED group) and 113 children from Standard 1-3 received normal classroom practice (PINK group) during the 8- week intervention period. Additionally, 90 children from Standard 2-3 received a tablet- based intervention that did not involve the Masamu software (BLUE group). This was a critical additional control group that was incorporated into the study design to differentiate the generic effects of using tablets over the specific effects of the EuroTalk Masamu software. Due to the limited numbers of children enrolled in Standard 1 at the start of the school year, Standard 1 children were not allocated to this Non- Masamu Tablet Intervention. For all of the analyses below, a matched sample was determined which included all children from Standard 1-3 (n = 283) that were present at both pre- test and post- test. Table 2 gives the descriptive statistics of this matched sample. 9

Table 1. Composition of the study sample at each stage of the RCT Study Phase Enrolment Eligible (All children in S1-4 attending school on first and second day of school year) Randomised (To one of three intervention groups) Excluded (Due to group size study design constraints) Allocation Intervention group Randomised to group Pretested Not pretested (i.e. absent at pre- test) Received intervention Did not receive intervention (i.e. transferred before pre- test) Follow up Post tested Lost to follow up (i.e. absent or transferred by post- test) Analysed Excluded from analysis (i.e. absent at pre- test or post- test or transferred) Matched sample (i.e. present at pre- test and post- test) Masamu Tablet 115 S1 = 25; S2 = 42 S3 = 48 112 S1 = 24; S2 = 41 S3 = 47 1 S1 = 0; S2 = 1 S3 = 0 113 S1 = 24; S2 = 42 S3 = 47 2 S1 = 1; S2 = 0 S3 = 1 105 S1 = 22; S2 = 39 S3 = 44 8 S1 = 2; S2 = 3 S3 = 3 11 (9.6%) S1 = 3; S2 = 4 S3 = 4 104 S1 = 22; S2 = 38 S3 = 44 Number of Children 436 (S1 = 73; S2 = 125; S3 = 152; S4 = 86) 400 (S1 = 49; S2 = 125; S3 = 144; S4 = 82) 36 (S1 = 24; S2 = 0; S3 = 8; S4 = 4) Non- Masamu Tablet 90 S1 = 0; S2 = 42 S3 = 48 83 S1 = NA; S2 = 37 S3 = 46 2 S1 = NA; S2 = 2 S3 = 0 85 S1 = NA; S2 = 39 S3 = 46 5 S1 = NA; S2 = 3 S3 = 2 82 S1 = NA; S2 = 38 S3 = 44 4 S1 = NA; S2 = 2 S3 = 2 11 (12.2%) S1 = NA; S2 = 7 S3 = 4 79 S1 = NA; S2 = 35 S3 = 44 Normal Practice 113 (*195) S1 = 24; S2 = 41 S3 = 48; *S4 = 82 109 (*190) S1 = 23; S2 = 40 S3 = 46; S4 = 81 3 (*4) S1 = 0; S2 = 1 S3 = 2; S4 = 1 112 S1 = 23; S2 = 41 S3 = 48 1 S1 = 1; S2 = 0 S3 = 0 103 S1 = 20; S2 = 38 S3 = 45 9 S1 = 3; S2 = 3 S3 = 3 13 (11.5%) S1 = 4; S2 = 4 S3 = 5 100 S1 = 20; S2 = 37 S3 = 43 * S4 children were tested at pre- test only to determine the typical developmental trajectory without intervention 10

Table 2. Descriptive statistics of the matched sample Age (mean (standard deviation) and min- max), gender (number of females F and males M), and handedness (number of left- handed L and right- handed R children) reported for each of the standards assessed Standard Intervention group Masamu ipad Learning Centre Non- Masamu ipad Learning Centre Normal Practice Classroom S1 (n = 44) n = 22 NA n = 20 Age (months) 84 (10.8) 73-118 81 (4.5) 74-93 Gender (F:M) 10:12 5:15 Handedness (L:R) 6:16 1:19 S2 (n = 110) n = 38 n = 35 n = 37 Age (months) 91 (8.7) 74-109 91 (9.5) 79-119 92 (10.7) 75-125 Gender (F:M) 18:20 19:16 17:20 Handedness (L:R) 8:30 7:28 2:35 S3 (n = 131) n = 44 n = 44 n = 43 Age (months) 106 (15.9) 87-161 109 (13.4) 87-147 105 (14.2) 79-139 Gender (F:M) 22:22 24:20 24:19 Handedness (L:R) 3:41 7:37 2:41 *S4 (n = 81) NA NA n = 81 **Age (months) 133 (11.8) 105-155 Gender (F:M) 45:36 Handedness (L:R) 6: 75 * S4 children were not allocated to intervention group and were tested at pretest only. ** Age (months) was not available for 5 children in S4. Descriptive statistics for Age are calculated for 76 children only. 11

3.3 Interventions Children from Standard 1-3 were randomly allocated to one of the following interventions. Masamu Tablet Intervention This intervention consisted of four different apps developed by EuroTalk : Masamu 1, Masamu 2, Count to 10, and Count to 20. The apps are based on the National Primary Curriculum that is delivered in Malawi and teach core mathematical concepts in a structured manner through several colourful and engaging sets of activities delivered in the local language, Chichewa. Children worked through the apps at their own pace and could practise particular activities as often as they desired. To progress to the next set of activities, children needed to pass a quiz built into the software that assessed knowledge of the set of activities the child had been working on. Teachers monitored progress of individual children through achievement charts, in which a star was awarded to each child as they passed a particular quiz. Passing a quiz required 100% accuracy, thus thoroughly assessing children s progression through the apps. Non- Masamu Tablet Intervention This intervention consisted of four different apps that are freely available to download from the Internet: Music Sparkles developed by Kids Game Club, Drawing Pad developed by Darren Murtha Design, and Toca Tailor and Toca Hair Salon developed by Toca Boca AB. These apps were chosen because they are educational (supporting musical ability and design skills), receive good customer ratings, are non- verbal, and do not involve concepts taught in the Masamu Tablet Intervention. Additionally, these apps require children to interact with the tablet in terms of manual and attentional processes in a similar manner to the Masamu apps. Throughout the intervention period, children were free to choose whichever app they wanted, as often as they wanted. 12

Normal Practice This intervention consisted of normal pedagogical practice delivered in primary schools across Malawi. Children followed the Malawi National Primary Curriculum, delivered in Chichewa by their classteacher, in the usual class setting. Basic numeracy (including mathematics) is taught from Standard 1. Location of Intervention: Both of the tablet- based interventions were administered in a purpose- built Unlocking Talent Learning Centre ; a small classroom within the grounds of Biwi Primary School but detached from the main school buildings (see Figure 2a). The Learning Centre housed up to 25 children at a time, sat on bamboo mats, individually using tablets connected to personal headsets (see Figure 2b). A classteacher was present in the Learning Centre during the tablet- based interventions primarily to assist the children with using the technology. Additional technical support was provided to classteachers throughout the intervention period by a volunteer from VSO. Classteachers responsible for overseeing the tablet- based interventions were given a training session prior to the intervention starting, in which they were shown how to use the tablets (turning the tablets on and off, connecting the headsets, storing the headsets (Figure 2c), charging and securely storing the tablets, and navigating through the software). They were also shown how to record attendance and progress with the Masamu software for individual children throughout the intervention period using attendance/achievement charts specifically designed for this study that were pinned to a board in the Learning Centre (Figure 2d). Children allocated to the Normal Practice intervention received the National Primary Curriculum in their usual classroom setting. 13

Figure 2. Images of the Unlocking Talent Learning Centre built at Biwi Primary School, Lilongwe (a) Outside of the Learning Centre (b) Masamu Tablet Intervention being delivered in the Learning Centre (c) Headphone rack constructed in the Learning Centre to ease set- up and storage during the tablet interventions (d) Daily attendance/achievement charts for the two tablet intervention groups in the Learning Centre 3.4 Assessments Tablet technology was used in the assessment of individual children on tasks of mathematical ability and associated basic skills as it enabled performance to be measured objectively, with a large sample of children, within a short period of time. Accordingly, a new assessment app was developed specifically for this study in which instructions were given in Chichewa. The app consisted of two measures of mathematical ability, six measures of basic skills known to be associated with scholastic progression, and a questionnaire measuring ambitions and attitudes. All of these measures were designed by the researcher (author of this report) and were programmed by EuroTalk. Figure 3 illustrates the different tasks. An introduction task in using the tablet was also included, which taught children the critical operations required to complete the tasks in the assessment app, including how to 14

select objects on the screen varying in size and how to select and move objects around the screen. Basic Skills Six measures of basic skills related to scholastic progression (see Wei et al., 2011) were developed to gain a deeper understanding of the nature of the underlying difficulties low achievers in mathematics might have. Three measures (Manual Processing Speed, Manual Coordination, and Visual Attention) were speeded so the software automatically recorded response time (milliseconds). The other three measures (Short Term Memory, Working Memory, and Spatial Intelligence) were not speeded so the software recorded accuracy of response. For each task, children were given a set of practice trials to familiarise themselves with the task demands, prior to the experimental trials commencing. The tasks assessing Short Term Memory, Working Memory, and Spatial Intelligence increased in difficulty with successive trials so a discontinue rule was applied to each task that terminated the task after a set number of consecutive fails. Mathematical Ability Two measures of mathematical ability were developed: 1. Maths Curriculum Knowledge (CK) consisted of 50 quiz items taken from the EuroTalk Masamu 1 and Masamu 2 apps and thus assessed curriculum knowledge specific to the Masamu Tablet Intervention. 2. Mathematical Concepts (MC) consisted of 48 questions assessing a conceptual understanding of mathematics, similar to that used in the USAID (2010) study and the Numerical Operations subtest of the WIAT- II (Wechsler, 2005). Concepts assessed in this task included symbolic understanding, numbers in relation to each other, number line understanding, counting, number sense (quantity estimation), simple and complex addition, simple and complex subtraction, and multiplication and division. For both of these measures items were presented in a set order that increased with difficulty over successful trials. Accordingly, a discontinue rule was applied so that the task terminated after a specified number of consecutive fails. This prevented children becoming demoralised by having to answer questions that were beyond their ability. 3. Maths Curriculum Knowledge Generalisation (CKG). This additional measure was given at post- test only to assess generalisation of curriculum knowledge learned through the tablet- based intervention to a more conventional paper and pencil context. Accordingly, EuroTalk developed 50 new items that were based on those used in the Masamu Tablet Intervention. Paper tests were prepared for each child, with their photograph and study number on the front page to ease identification of individual children. These were administered in groups of 25 children in the Learning Centre, with an assistant pointing to each question as a classteacher read out the question in Chichewa. 15

Ambitions and Attitudes 9) An Ambitions and Attitudes Questionnaire (AAQ) was developed to assess life ambitions and attitudes towards learning. It consisted of 10 questions, given in Chichewa through the tablet- based assessment app, which required choosing one response from five pictorial options provided. Questions measuring attitudes required children to use a happy- sad pictorial likert scale to express their response by choosing one of the five faces presented that best matched how they felt (Reynolds- Keefer et al., 2009). Children were told there were no right or wrong answers to these questions and that they should choose the picture that best matched their answer. Questions 1-4 assessed ambitions through the following questions: 1. How many babies do you think you will have? 2. How much money do you think you will earn? 3. What job do you think you will do? 4. When do you think you will leave school? Questions 5-10 assessed attitudes towards school and learning through the following questions: 5. When you are at home how do you feel? 6. When you are at school how do you feel? 7. When you play with these gadgets how do you feel? 8. When you play with your friends how do you feel? 9. When someone hurts you how do you feel? 10. When someone gives you a present how do you feel? Questions 9 and 10 served as control questions to measure the extent to which children were responding reliably on this task. 16

Figure 3. Illustrations of the tasks included in the assessment app Task Order Function Measured Task stimuli & task instructions 1 Manual Processing Speed (MPS) Tap the green square as fast as you can 2 Manual Coordination (MCO) There are two green squares. Touch one square then the other as fast as you can 3 Short Term Memory (STM) Copy me 4 Visual Attention (VA) Touch the pink dot. Touch all the pink dots as fast as you can. Remember touch just the pink dots 5 Working Memory (WM) Watch me. You do it backwards 6 Spatial Intelligence (SIQ) Look at this pattern. Use these blocks. Make the same pattern here 17

Figure 3 cont. Illustrations of the tasks included in the assessment app Task Order Function Measured Task stimuli & task instructions 7 Maths Curriculum Knowledge (CK) Cross out the odd one out How many footballs are there? Touch the number Put four tomatoes on the plate 8 Mathematical Concepts (MC) Touch all of the numbers in the box Touch the biggest number Drag the numbers in order to the spaces below 9 Ambitions and Attitudes Questionnaire (AAQ) How many babies do you think you will have? What job do you think you will do When you play with these gadgets how do you feel? 3.5 Procedure The study commenced in September 2013 on the first day of the school year and continued for 10 weeks. Enrolment and pretesting was carried out over the first week by the researcher (the author of this report) and three assistants (a volunteer from VSO and a programmer and a translator from EuroTalk ). The intervention took place over the following 8 weeks and was implemented by classteachers at Biwi Primary School. Technical support was provided by the VSO volunteer. Post- test assessments were then conducted in November 2013, in week 10, by the researcher and two assistants (the volunteer from VSO and the programmer from EuroTalk ). Enrolment: Over the first two days of the school year all children that were eligible for the study were enrolled, randomly allocated to one of the three intervention groups, and assigned a study number. Enrolment included photographing each child so as to ease identification of individual children assigned to the different intervention groups. The researcher assessed handedness by presenting each child with a pencil to their midline and asking them, in Chichewa, to take it. The hand 18

the child used to take the pencil from the researcher was recorded as their preferred hand. Gender, Standard, and Classroom were also recorded for each child and date of birth was collected through school records. To ensure children were enrolled just once, a temporary mark was placed on the child s hand to indicate they had been registered. A computer program, developed by EuroTalk, was used to automatically assign children randomly to one of the three intervention groups. Figure 4. Researcher and assistant enrolling a child for the study Pre- test and post- test assessments: Groups of up to 50 children maximum were pre- tested or post- tested at the same time, in a regular classroom in the main school building. Prior to assessment, the tablets were set up with each child s photograph, study number, and intervention group. This aided conducting the pre- test and post- test assessments as the tablets could be handed out reliably to individual children and their data was stored automatically to their allocated intervention group. Children were tested per intervention group (e.g. Standard 1 RED) where possible. Children were collected from their classroom by one of the assistants and escorted to the classroom where the assessments were carried out. When entering the assessment classroom the researcher or assistant handed each child the tablet set up specifically for them and a headset. Children sat on the floor to carry out the assessments (Figure 5a). The researcher stood at the front of the group and demonstrated each task in the assessment app via a large tablet with instructions being orally translated by a classteacher. Children first completed a brief demonstration of how to touch and move objects on the tablet screen, so as to familiarise themselves with using the technology (Figure 5b). They then carried out each task in the assessment app, in the order specified above. To start each task children were required to swipe a large white dot on an introduction screen (Figure 5c) after which the first practice trial of that task was administered. Children worked through each task, in sequence, until they had completed all of the assessments. A large star appearing on the screen and exploding into smaller stars marked the end 19

of the assessment app (Figure 5d). Children took around 40-60 minutes to complete the assessment app, depending on their level of ability. Upon completion, children handed their tablet and headset to the researcher or assistants and returned to their class. Figure 5. Images of the pre- test and post- test assessments (a) Children being assessed at post- test (b) Example from the ipad demonstration task (c) Example of an introduction screen to each task in the assessment app (d) Example of a completion screen marking the end of the assessment app Interventions: The intervention period lasted 8 weeks (40 school days). Each of the tablet- based interventions was administered for 20 days. Children from Standard 2-3 received the tablet- based intervention to which they had been allocated for one hour on alternate school days (20 hours maximum across the 8- week intervention period). Children from Standard 1 received the Masamu Tablet Intervention for 30 minutes on alternate school days (10 hours maximum across the 8- week intervention period). Classteachers responsible for overseeing the tablet- based interventions drew up a timetable, in which one set of teachers was allocated to the Masamu intervention (RED group) and a second set of teachers was allocated to the Non- Masamu intervention (BLUE group). The tablets used in this study were ipad Minis as these are a suitable size for young children to hold and interact with whilst seated on the floor. EuroTalk placed 50 ipads into Biwi Primary School specifically for this study. This enabled 25 ipads to be 20

used for the intervention on alternate days whilst the other 25 ipads were being charged. To ensure children received the correct intervention, 25 ipads with RED covers had only the Masamu intervention software installed on them, and the other 25 ipads with BLUE covers had only the Non- Masamu software installed on them. Furthermore, for the Masamu intervention group ipads were personalised to individual children using their photographs and study number allocated at enrolment. This ensured that each child used the same ipad throughout the intervention period and allowed electronic data to be recovered on individual usage. Classteachers overseeing the tablet- based interventions were responsible for charging the ipads when not in use. Time on task was recorded for individual children through attendance registers either taken in the Learning Centre (for the tablet- based interventions) or the class register for the Normal Practice control group. For the Masamu Tablet Intervention, teachers tracked progress by awarding stars on an achievement chart to children when they passed a quiz marking successful completion of a set of activities. (Note: the electronic information stored in the ipads could also be used to track individual progress and to crosscheck teachers recording of progress on the achievement charts.) So as to equate the learning experience of the tablet- based interventions as far as possible, stars were also awarded on a chart in the Learning Centre to children in the Non- Masamu Tablet Intervention when they attended a session. 3.6 Data analysis For all of the analyses below, the matched sample reported in Table 2 above was used. Mean performance was determined for individual children for each of the tasks in the assessment app, except for the Ambitions and Attitudes Questionnaire where frequency of response was determined for each of the intervention groups. To address the three principle and three secondary aims of this study the following analyses were conducted. 1) To evaluate the effectiveness of the EuroTalk Masamu Tablet Intervention over normal classroom practice in supporting the development of mathematical ability, mean performance for each of the intervention groups was compared across time, at both pre- test and post- test, using mixed Analyses of Variance (ANOVA), for Standard 1-3. Separate 3 (Intervention Group: Masamu Tablet Intervention, Non- Masamu Tablet Intervention, and Normal Practice) x 2 (Time: pre- test and post- test) mixed ANOVAs were conducted for Standard 2 and Standard 3 and a 2 (Intervention Group: Masamu Tablet Intervention and Normal Practice) x 2 (Time: pre- test and post- test) mixed ANOVA was conducted for Standard 1. 2) To establish the most appropriate Standard in which to implement the EuroTalk Masamu Tablet Intervention, effect sizes (using Cohen s d) were computed for each Standard at post- test for the Masamu Tablet Intervention group compared to the Normal Practice control group. Effect sizes provide a standardized measure on which to compare the magnitude of the intervention effect thus enabling direct comparison across Standards. According to Cohen (1988) an effect size of 0.2 is considered small, 0.5 is considered medium, and 0.8 or above is considered large. 21

3) To determine if girls respond differently to the EuroTalk Masamu Tablet Intervention than boys, mean performance for the combined measure of mathematical ability of girls and boys receiving the Masamu Tablet Intervention was compared to that of girls and boys receiving Normal Practice at pre- test and post- test, using a 2 (Gender) x 2 (Intervention Group) x 2 (Time) mixed ANOVA. 4) To ascertain the basic skills related to mathematical ability, each measure of basic skill included in the assessment app was correlated (using Partial Correlations, controlling for age) with the two measures of mathematical ability (MC and CK), for each intervention group. Negative correlations were predicted with the speeded measures of basic skill (MPS, MCO, VS). Positive correlations were expected with the accuracy measures of basic skill (STM, WM, SIQ). 5) To identify low achievers in mathematical ability and the associated basic skills underpinning their abilities, group mean performance and the standard deviation (sd) was determined for each Standard on the combined measure of mathematical ability, then children falling 2 sds below the group mean were identified statistically as low achievers. Their profile of performance on the basic skills associated with scholastic progression was then inspected to establish specific areas of weakness (i.e. basic skills where performance fell below 2 sds of the group mean). Finally, response to the intervention received was investigated in the group of low achievers by comparing their performance at post- test to the average level attained by their Standard. 6) To investigate if the EuroTalk Masamu Tablet Intervention altered children s ambitions and attitudes towards learning compared to normal classroom practice over the 8- week intervention period, the most frequent response given to each of the questions asked in the Ambitions and Attitudes Questionnaire was compared for children receiving the Masamu Table Intervention to those receiving Normal Practice at pre- test and post- test using a series of chi square tests. 22

4.0 RESULTS Summary statistics of the analyses performed are reported in Appendices I- V. 1) Effectiveness of the EuroTalk Masamu Tablet Intervention: Table 3 reports the mean performance at pre- test and post- test for each of the intervention groups across Standard 1-3 on the two measures of mathematical ability (MC and CK) included in the tablet- based assessment app and the paper and pencil generalisation test (CKG) given at post- test. As can be seen, for each Standard, greater gains in performance across time were achieved by the Masamu Tablet Intervention group compared to the other intervention groups, for all measures of mathematical ability. Key results from the ANOVAs conducted for each Standard (see Appendices I- III) revealed the Masamu Tablet Intervention group showed: Significantly greater gains in performance over time compared to the Normal Practice control group on the Mathematical Concepts (MC) test for Standard 1 and Standard 2 children. Significantly greater gains in performance over time compared to both the Non- Masamu Tablet Intervention group and the Normal Practice control group on the Maths Curriculum Knowledge (CK) test for Standard 2 and Standard 3 children. Significantly higher performance on the paper and pencil generalisation test (CKG) at post- test than the Non- Masamu Tablet Intervention group for Standard 2 children, and both the Non- Masamu Tablet Intervention group and the Normal Practice control group for Standard 3 children. These results suggest that intervention with the EuroTalk tablet- based Masamu software is more effective than normal pedagogical practice in Malawi at supporting development of mathematical skills in primary school children. Dedicated time spent interacting with tablet technology in small groups away from the rest of the class is insufficient to support mathematical skills, as the gains made by the Non- Masamu Tablet Intervention group did not differ significantly from the Normal Practice control group. Intervention with the EuroTalk tablet- based Masamu software seems to support initially a conceptual understanding of mathematics in Standard 1 and 2 and then more specific curriculum- based knowledge in Standard 2 and 3, which generalises to different formats (as shown by the paper and pencil test). 23

Table 3. Intervention group performance at pre- test and post- test Pre and post intervention performance (mean (standard deviation), min- max) and percentage performance gain for the tests of Mathematical Concepts (MC) (max score 48), Maths Curriculum Knowledge (CK) (max score 50), and Maths Curriculum Knowledge Generalisation (CKG) (max score 55) Standard Intervention group Assesment Time of test Masamu ipad Learning Centre Non- Masamu ipad Learning Centre Normal Practice Classroom S1 (n = 44) n = 22 NA n = 20 MC Pre 2.0 (1.7) 0-5 3.4 (4.3) 0-13 Post 5.1 (4.6) 0-17 4.5 (4.3) 0-13 % gain 6.5 2.3 CK Pre 2.4 (2.7) 0-12 2.7 (1.7) 0-6 Post 7.7 (6.3) 0-24 5.9 (5.3) 0-20 % gain 10.6 6.4 S2 (n = 110) n = 38 n = 35 n = 37 MC Pre 8.6 (6.5) 0-20 10.1 (5.7) 0-21 8.4 (6.3) 0-19 Post 14.6 (6.6) 0-24 12.5 (7.3) 0-23 10.5 (6.5) 0-19 % gain 12.5 5.0 4.4 CK Pre 6.4 (6.3) 0-24 8.3 (8.1) 0-28 5.5 (5.3) 0-21 Post 20.7 (10.3) 0-37 15.1 (8.5) 2-31 10.8 (7.4) 0-26 % gain 28.6 13.6 10.6 CKG Post 23.8 (9.1) 7-41.5 17.4 (10.3) 0-36 20.7 (8.4) 4-39 S3 (n = 131) n = 44 n = 44 n = 43 MC Pre 14.9 (5.7) 0-23 15.4 (5.6) 0-22 14.9 (5.2) 0-22 Post 19.5 (5.2) 5-27 19.3 (5.1) 4-26 18.6 (6.0) 0-26 % gain 9.6 8.1 7.7 CK Pre 13.5 (9.3) 0-34 14.8 (10.1) 1-37 11.0 (10.3) 0-33 Post 35.2 (7.0) 19-46 24.9 (8.0) 3-36 23.4 (6.9) 7-36 % gain 43.4 20.2 24.8 CKG Post 33.5 (7.3) 19-47.5 20.0 (6.7) 7-36 27.2 (8.1) 10-43 *S4 (n = 81) NA NA n = 81 MC Pre 20.5 (4.6) 0-33 CK Pre 18.8 (11.4) 0-36 * S4 children were not allocated to intervention group and were tested at pretest only. 24

2) Most appropriate Standard in which to implement the EuroTalk Masamu Tablet Intervention: Results from the analyses above suggest that it is most appropriate to implement the EuroTalk Masamu Tablet Intervention in Standard 2 as the Masamu Tablet Intervention significantly supported both a conceptual understanding of mathematics and specific curriculum- based maths knowledge in this Standard. In general, this result was supported by the effect size analyses (Table 4) conducted at post- test across the Masamu Tablet Intervention group and the Normal Practice control group. As Table 4 shows, Standard 2 children showed the largest effect of the Masamu Tablet Intervention for the Mathematical Concepts (MC) test compared to Standard 1 and Standard 3 children. Standard 2 children also showed a very large effect of the Masamu Tablet Intervention for the Maths Curriculum Knowledge (CK) test although the effect size was even larger for Standard 3 children for this measure and Standard 3 showed the largest effect size for the paper and pencil generalisation test (CGK). Table 4. Effect size analyses Effect sizes (Cohen s d) across Standard 1-3 for differences in mean performance of the Masamu Tablet Intervention group compared to the Normal Practice control group at post- test for the Mathematical Concepts (MC), Maths Curriculum Knowledge (CK), and Maths Curriculum Knowledge Generalisation (CKG) test Standard Measure MC CK CKG 1 0.135 0.310 NA 2 0.626 1.119 0.354 3 0.161 1.698 0.818 Overall, these results suggest that implementing the EuroTalk Masamu Tablet Intervention in Standard 2 is optimal for supporting development of both conceptual and specific mathematical abilities. However, Standard 3 children benefit most from the EuroTalk Masamu Tablet Intervention in supporting development of specific maths knowledge relating to the National Primary Curriculum in Malawi. [Note: When comparing effect sizes across Standards it is important to note that children in Standard 1 received 20 hours of intervention with the EuroTalk Masamu software whereas children in Standard 2 and 3 received 40 hours. Although the effect sizes in Standard 1 were small (MC) to small- medium (MQ) it is likely that larger effects of Masamu Tablet Intervention could be gained by increasing the time on task in Standard 1.] 3) Difference across girls and boys to the EuroTalk Masamu Tablet Intervention: Figure 6 shows the mean performance gains for girls and boys across Standard 1-3 for the Masamu Tablet Intervention group and the Normal Practice control group on 25

a combined measure of mathematical ability (average performance across the MC and CK tests /49). As is clearly shown, there are no noticeable differences in performance gains across gender. This was confirmed by the ANOVA, which showed no significant main effect of gender, and no significant interactions with gender across time and/or intervention group (see Appendix IV). Separate ANOVAs conducted on the MC and MQ measures of mathematical ability confirmed no significant gender effects in this sample. Overall, these results suggest that both girls and boys responded similarly to each intervention, showing that the EuroTalk Masamu Tablet Intervention is just as effective for girls as it is for boys. Figure 6. Gender histogram Differences in the percentage gain over time for girls and boys across Standard 1-3 for the Masamu Tablet Intervention group and the Normal Practice control group on a combined measure of maths ability 4) Basic skills related to mathematical ability: Appendix V reports the partial correlations, controlling for age, between the six different measures of basic skill and the two measures of mathematical ability at pre- test and post- test, for the whole sample, and the three intervention groups separately. As expected, the two measures of mathematical ability (MC and MQ) were positively correlated, and strength of correlation increased over the intervention period. This suggests that children s conceptual understanding of mathematics and their curriculum specific knowledge became more integrated over the 8- week intervention period, particularly for the Masamu Tablet Intervention group who showed the strongest correlation at post- test. 26

As predicted, the speeded measures of basic skill (MPS, MCO, VS) produced negative correlations with mathematical ability (except for two correlations with MCO), illustrating that children who can execute basic manual and attentional tasks quickly are likely to have better maths abilities than those who are slow on these tasks. Likewise, as expected, the accuracy measures (STM, WM, SIQ) produced positive correlations with the measures of mathematical ability (apart from two correlations with WM), illustrating that children with good memory skills and good spatial intellience are likely to be good at maths. Comparing across tasks, the measures of working memory (WM) and manual coordination (MCO) correlated least consistently with the two measures of maths ability, whereas the measures of short term memory (STM), spatial intelligence (SIQ), manual processing speed (MPS) and visual attention (VA) produced the most consistent correlations. These results corroborate previous research with Western populations (see Wei et al., 2011) and suggest that these basic skills form the foundation of the development of mathematical ability in primary school children. Reliability and validity of the new assessment app: As tablet technology was used to assess basic skills in this study by the development of a new assessment app, it is important to establish the reliability and validity of this new assessment app at measuring mathematical ability and basic skills. To establish the reliability of the assessment app, test- retest correlations were conducted (using Pearson s Product Moment Correlation Coefficients) with the sample of children in the Normal Practice control group, as these children were not exposed to the tablet technology across the intervention period. Table 5 summaries the results and shows that all measures included in the new assessment app correlated significantly and positively over time, except for the measures of manual processing speed (MPS) and manual coordination (MCO) where the correlations were positive but not significant. This could be that these two tasks were the first to be administered and the children were very excitied about using the ipads so attention levels could have varied considerably. Overall, these results indicate that the new assessment app is a reliable method for assessing basic skills and mathematical abilities in primary school children in Malawi. 27

Table 5. Reliability of the new assessment app Test- retest reliability correlations for the ipad assessments over the 8 week study period with the normal classroom practice control group (n = 100) Measure Speeded tests Manual Processing Speed (MPS) Manual Coordination (MCO) * Visual Attention (VA) Accuracy tests Short Term Memory (STM) * Working Memory (WM) * Spatial Intelligence (SIQ) * Mathematical Concepts (MC) Maths Curriculum Knowledge (CK) Correlation Coefficient r.12.20.47.54.47.54.68.59 Significant correlations are highlighted in bold * S1 children were not given these tasks at pretest so n = 80 (for STM n = 79 as one child failed to complete this task) To assess the validity of the new assessment app, 27 Foundation Year children in a UK primary school were given the new assessment app and two sub- tests from a well- established, standardised, clinical measure of intelligence (WPPSI- IV, Wechsler, 2012). Performance on the new assessment app and the standardised clinical tests was correlated using Pearson s Product Moment Correlation Coefficients. Results are reported in Table 6. As predicted, all speeded measures correlated negatively with standard scores for Symbol Search and Block Design from the WPPSI although these correlations failed to reach significance, except for Block Design and the tablet measure of Manual Coordination (MCO). Interestingly, both of these tasks require manual manipulation to execute them successfully, so the signifcant correlation between these tasks indicates the new tablet- based measure of Manual Coordination (MCO) is valid. Block Design also correlated significantly with the tablet measure of Spatial Intelligence (SIQ) which is to be expected as the SIQ measure is a 2D version of the 3D Block Design task. Symbol Search correlated significantly with the tablet assessments of Short Term Memory (STM) and Working Memory. Additionally, Symbol Search and Block Design correlated significantly with the two tablet- based measures of maths ability. Overall, the new assessment app seems to be a valid method for assessing basic skills and maths ability in primary school children but requires further refinement of some measures (especially speeded measures) to improve its validity. 28

Table 6. Validity of the new assessment app Correlation coefficients (r) measuring the validity of the ipad assessments compared to standardised clinical tests (Symbol Search and Block Design from the WPPSI- IV) from a UK sample (n = 27) of Foundation Year children ipad Assessment Measure Standardised Measure Symbol Search Block Design r r Speeded tests Manual Processing Speed (MPS) -.30 -.24 Manual Coordination (MCO)* -.26 -.40 Visual Attention (VA) -.24 -.26 Accuracy tests Short Term Memory (STM).46.15 Working Memory (WM).55.17 Spatial Intelligence (SIQ).38.42 Mathematical Concepts (MC).69.52 Maths Curriculum Knowledge (CK).37 **.53 Significant correlations (2- tailed) are highlighted in bold * For this measure n = 26 * * p =.058 5) Basic skills profile of low achievers in mathematical ability: This analysis was conducted with children from Standard 2-3 only as Standard 1 children did not receive the full assessment app at pre- test. Standard 2: Of the 110 Standard 2 children in the matched sample 17 (15.5%) fell below 2 sds from the sample mean on the combined measure of mathematical ability (average of the MC and MQ tests) at pre- test. Of these, 8 had been randomly allocated to the Masamu Tablet Intervention, 4 to the Non- Masamu Tablet Intervention, and 5 to the Normal Practice control group. These 17 children thus performed significantly below the level expected for their Standard. At post- test, 8 (47%) of these 17 children performed either above (3 children) or within 1 sd below the sample mean (5 children), showing that their performance at post- test was within the normal level expected for their Standard. Of these 8 children, 6 (75%) had received the Masamu Tablet Intervention, 1 (25%) had received the Non- Masamu Tablet Intervention, and 1 (20%) had received Normal Practice. Each of the 3 children that improved to a level above the sample mean at post- test had received the Masamu Tablet Intervention. Inspection at pre- test on the performance of the basic skill tasks revealed specific areas of weakness in 5 (30%) of the 17 children with significantly poor mathematical ability (4 children had been allocated to the Masamu Tablet Intervention and 1 child had been allocated to the Normal Practice control group). Of these 5 children the child that had been allocated to the Normal Practice control group had specific weaknesses in Manual Coordination and Visual Attention and continued to perform 29

significantly below expected levels for mathematical ability at post- test. Of the 4 children that had been allocated to the Masamu Tablet Intervention, one child had a specific weakness in Visual Attention and continued to perform significantly below the expected level on mathematical ability at post- test, whereas the other three children performed within the normal range for mathematical ability at post- test, despite at pre- test one child showing a specific weakness in Manual Coordination and two children showing specific weaknesses in Short Term Memory. Standard 3: Of the 131 Standard 3 children in the matched sample, 4 (3%) performed significantly below the level expected for their Standard at pre- test on the combined measure of mathematical ability. At post- test, one child continued to perform significantly below the level expected in maths for their Standard; this child had received Normal Practice over the intervention period. The other 3 children all raised their maths performance to within 1 sd of the sample mean at post- test; 2 children had been allocated to the Non- Masamu Tablet Intervention and 1 child received the Masamu Tablet Intervention. Specific areas of weakness were identified at pre- test in 1 of these 4 children. This child who had been randomly allocated to the Masamu Tablet Intervention had a specific weakness in Manual Processing Speed at pre- test but raised their maths performance to within the normal level expected for their Standard at post- test. Overall, these results show that some children who significantly underachieve in maths have specific areas of weakness, but despite this they can overcome these difficulties to raise attainment levels in maths, especially if they have received the EuroTalk Masamu Tablet Intervention. [Note: Due to low sample mean performance and large variation on the measures of Working Memory and Spatial Intelligence neither of these tests were suitable to identify areas of specific weakness in Standard 2-3.] 6) Changes in ambitions and attitudes towards learning following the EuroTalk Masamu Tablet Intervention: A short questionnaire designed to measure children s ambitions and attitudes towards learning was given at pre- test and post- test. Two questions (9 & 10) were included as control questions to test if children were responding as expected. Initial inspection of the data gathered from across the entire matched sample revealed a high level of inconsistent responding to these two control questions, with more than 50% of children responding in the opposite direction to that expected (see Appendix VI). To improve reliability of this measure children were excluded from analyses who responded in the opposite direction to that expected on these two control questions, at both pre- test and post- test. This reduced the sample for analysis to just 26 children (19 in the Masamu Tablet Intervention and 7 in the Normal Practice). Results are reported in Table 7 below. 30

Table 7. Pre and post intervention performance on the Ambitions and Attitudes Questionnaire (AAQ) Most frequent response (given as a percentage) provided for each question of the AAQ at pre- test and post- test for the Masamu Tablet Intervention group compared to the Normal Practice, based on sample of 26 children that responded accurately to the control questions 9 & 10 AAQ Intervention group question Masamu Tablet (n = 19) Normal Practice (n = 7) Pre- test Post- test Pre- test Post- test Answer % Answer % Answer % Answer % Babies Two 47 Two 53 Two 100 Two 71 Money Lots 68 Lots 95 Quite 57 Lots 86 a lot Job Pilot 53 Pilot 63 Doctor & 43 Doctor 57 Pilot Leave Young 79 Young 100 Young 71 Young 100 school adult adult adult adult Home Very 37 Very 53 Very 71 Neutral 43 Happy Happy Happy School Very 47 Very 47 Very 57 Happy 43 Sad Happy Happy Gadgets Very 42 Happy 37 Neutral 43 Happy 43 Happy Friends Very Happy 42 Happy & Very Happy 42 Happy 57 Very Happy 71 As can be seen there was considerable consistency in responses generated on this measure across both intervention groups. Across the 8- week intervention period, no noticeable group differences were identified at either pre- test or post- test, except for the question which asked When you are at school how do you feel? where the Masamu Tablet Intervention group shifted their responding over the intervention period from Very Sad at pre- test to Very Happy at post- test. This difference was not significant however when Yates correction was applied (Yates χ2 = 4.87, p =.09). Overall, responses to this questionnaire over the 8- week intervention period appeared to be independent of intervention given and showed that most children hoped to have two babies, earn lots of money, have a professional career, leave school as young adults, and were generally happy at home and at school. 31

5.0 CONCLUSIONS (1) This RCT has shown clearly that the EuroTalk Masamu Tablet Intervention is significantly more effective at improving mathematical attainment than current pedagogical practice for primary school children in Malawi. (2) After just 8 weeks of using the EuroTalk Masamu Tablet Intervention children tripled their specific maths curriculum knowledge, with Standard 2-3 children raising attainment levels to a higher level than the average shown by Standard 4 with normal pedagogical practice (see Figure 7 below). Figure 7. Developmental trajectory of maths curriculum knowledge for the whole sample at pre- test (without intervention) and at post- test following the Masamu Tablet Intervention (3) The EuroTalk Masamu Tablet Intervention is most effective at supporting: (i) Conceptual understanding of mathematics when implemented in Standard 2 (ii) Specific maths curriculum knowledge when implemented in Standard 3. (4) The EuroTalk Masamu Tablet Intervention is just as effective at supporting the development of mathematical skills in girls as it is for boys. (5) Basic skills known to underpin scholastic attainment in Western populations correlate with maths ability in Malawian primary school children, suggesting these basic skills form the foundation of maths attainment in the primary years and could be used to predict mathematical development prior to and during formal instruction. 32

(6) Assessment of basic skills and mathematical ability using tablet- based technology is a reliable and valid method for objectively measuring performance in primary school children. However, further refinement of the new assessment app is needed to make it a more sensitive diagnostic tool. (7) The EuroTalk Masamu Tablet Intervention is effective in raising mathematical standards even in low achievers. Overall, 10% of Standard 2-3 children were identified as significantly low achievers in mathematical ability at pre- test, of which 1 in 4 were found to have specific weakness in at least one or more basic skills. Over the 8- week intervention period, 78% of low achievers who received the Masamu Tablet Intervention improved their maths ability to a level typical for their Standard, whereas only 17% of children who received Normal Practice raised their maths attainment to within the normal range. (8) Over the 8- week intervention period, children s ambitions and attitudes towards learning appeared to be independent of the intervention they received. Overall, children in the early primary years in Malawi generally reported to be happy with their home and school life and had ambitions to stay in school until young adulthood, pursue a professional career, and to have just two children. An effect of intervention on ambitions and attitudes towards learning might be seen however if the Masamu Tablet Intervention is administered for a longer period of time. 33

6.0 FUTURE CONSIDERATIONS This study evaluated the effectiveness of the EuroTalk Masamu Intervention in supporting acquisition of early mathematical skills in primary school children in Malawi over an 8- week period. Although the intervention period lasted just 8 weeks, significant effects of the intervention were found compared to normal pedagogical practice. Further research is needed to establish the following: Evaluating the long- term effectiveness of the EuroTalk Masamu Intervention when implemented over a full school year and when embedded within normal pedagogical practice. Generalisation to other contexts within Malawi by comparing the effectiveness of the EuroTalk Masamu Intervention across schools varying in key demographics (e.g. rural versus urban schools, schools with electricity compared to schools without electricity, schools from rich areas compared to schools from poor areas, schools where attainment levels are high compared to schools where attainment levels are low etc). Generalisation to other countries by comparing the effectiveness of the EuroTalk Masamu Intervention across countries with different pedagogies and curriculums through scaffolding learning by giving instructions in the child s local language. Optimising learning with the EuroTalk Masamu Intervention through (i) refinement of the software design and content and (ii) comparing methods of implementation (e.g. individual versus didactic interaction, peer- tutoring versus scaffolding by classteachers, varying time on task, Learning Centre versus normal classroom setting etc). Identifying the infrastructure and technical support needed to ensure the EuroTalk Masamu Intervention is implemented appropriately and usage is sustained over time, as hardware and software upgrades become available, and hardware needs to be replaced. This will have substantial financial implications that need to be considered for sustainable use. Determining the training needs of teachers to enable them to implement the EuroTalk Masamu Intervention confidently and appropriately within their everyday practice, so as to ensure that every child who receives the intervention does so in a way that will maximise learning. Refining and further development of the new technology- based assessment app to make it a more sensitive diagnostic tool for identifying children at risk of learning difficulties. 34

7.0 REFERENCES Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Means, B., Murphy, R., Penuel, W., Javitz, H., Emery, D., & Sussex, W. (2007). Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort, Washington, D.C.: U.S. Department of Education, Institute of Education Sciences. Duncan, G.J., et al. (2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428 1446. Ke, F. (2008). A case study of computer gaming for math: Engaged learning from gameplay? Computers & Education, 51(4), 1609-1620. Lipton, J. S., & Spelke, E. S. (2003). Origins of number sense: large- number discrimination in human infants. Psychological Science, 14(5), 396 401. Reynolds- Keefer, L., Johnson, R., Dickenson, T., McFadden, L. (2009). Validity issues in the use of pictorial likert scales. Studies in Learning, Evaluation, Innovation, and Development, 6(3), 15-24. Rosas, R., Nussbaum, M., Cumsille, P., Marianov, V., Correa, M., Flores, P., et al. (2003). Beyond nintendo: Design and assessment of educational video games for first and second grade students. Computers and Education, 40(1), 71 94. Sandford, R., Ulicsak, M., Facer, K. & Rudd, T. (2006). Teaching with Games: Using commercial off- the- shelf computer games in formal education. Bristol: Futurelab. Siegler, R.S., Duncan, G.J., Davis- Kean, P.E., Duckworth, K., Claessens, A., Engel, M., Susperreguy, M.I., & Chen, M. (2012). Early predictors of high school mathematics achievement. Psychological Science, 23(7), 691-697. Wechsler, D. (2005). Wechsler Individual Achievement Test 2nd Edition (WIAT II). London: The Psychological Corp. Wechsler, D. (2012). Wechsler Preschool and Primary Scale of Intelligence 4th Edition (WPPSI IV). London: The Psychological Corp. Wei, W., Yuan, H., Chen, C., & Zhou, X. (2011). Cognitive correlates of performance in advanced mathematics. British Journal of Educational Psychology, 82(1), 157-181. Yelland, N., & Kilderry, A., (2010). Becoming numerate with information and communications technologies in the twenty- first century. International Journal of Early Years Education, 18(2), 91 106. 35

8.0 ACKNOWLEDGEMENTS Thanks and appreciation goes to the following people that kindly supported this study: EuroTalk Andrew Ashe (Managing Director) Richard Howeson (Chairman) Jamie Stuart (Chief Programmer) Michal Safier (Programmer) Alexandra Turner (Translator) VSO Malawi Dario Gentili (Country Director) Frederick Gaghauna (Technical Support) Malawi Ministry of Education Anjimile Oponyo (Principal Secretary for Basic and Secondary Education) Raphael Agabu (Director of Inspectorate for Schools) Henry Mphiningo (Primary Education Advisor Area 36) Dunkirk Primary School, Nottingham Foundation Year teachers and Laura Outwaite who provided the validation data for the new assessment app Kumbali Lodge, Lilongwe Guy and Maureen Pickering (Propriators) Special thanks to Guy for building the Learning Centre at Biwi Primary School Scottish Government For funding this study Special thanks to Biwi Primary School Special thanks to the Headteacher, Deputy Headteachers, and classteachers who worked tirelessly to support this study and to implement the interventions within their school. Enormous thanks also go to all of the Standard 1-4 children that participated in this study with such compliance and enthusiasm. You were a joy to work with. 36

9.0 APPENDICES Appendix I: Standard 1 ANOVA summary tables (i) Mathematical Concepts: 2 (Time) x 2 (Intervention group) mixed ANOVA Intervention group Mathematical Concepts n mean (sd) Pre Post Masamu Tablet 22 2.0 (1.7) 5.1 (4.6) Normal Practice 20 3.4 (4.3) 4.6 (4.2) Results Significant main effect of time F (1,40) = 7.2, p =.011 No significant main effect of group ns F (1,40) =.189, p =.666, ns No significant interaction between time and group ns F (1,40) = 1.71, p =.198 Planned comparisons Masamu group made significant gains over time t (21) = 3.11, p =.005 Normal Practice control group did not make significant gains over time t (19) =.89, p =.386 (ii) Maths Curriculum Knowledge: 2 (Time) x 2 (Intervention group) mixed ANOVA Intervention group Maths Curriculum Knowledge n mean (sd) Pre- test Post- Test Masamu Tablet 22 2.4 (2.7) 7.7 (6.3) Normal Practice 20 2.7 (1.6) 5.9 (5.3) Results Significant main effect of time F (1,40) = 24.15, p <.001 No significant main effect of group F (1,40) =.524, p =.473, ns No significant interaction between time and group F (1,40) = 1.63, p =.21, ns Planned comparisons Both groups make significant gains over time Masamu group t(21) = 4.17, p <.001 Normal Practice control group t(19) = 2.77, p =.012 37

Appendix II: Standard 2 ANOVA summary tables (i) Mathematical Concepts: 2 (Time) x 3 (Intervention group) mixed ANOVA Intervention group Mathematical Concepts n mean (sd) Pre Post Masamu Tablet 38 8.6 (6.5) 14.6 (6.6) Non- Masamu Tablet 35 10.1 (5.7) 12.5 (7.3) Normal Practice 37 8.4 (6.3) 10.5 (6.5) Results Significant main effect of time F (1, 107) = 32.13, p <.001 No significant main effect of group F (2, 107) = 1.54, p =.219 Significant interaction between time and group F (2, 107) = 4.13, p =.019 Analysis of simple main effects Pre- test: no significant effect of group F (2, 107) =.82, p =.443 Post- test: significant effect of group F (2,170) = 3.32, p =.039 Post- hoc tests with Bonferroni Tablet Control and Normal Practice p =.657, ns Masamu and Tablet Control p =.585, ns Masamu sig > Normal Practice p =.033 Masamu intervention group showed sig. better performance over time than normal practice control group Over time, Masamu intervention group and normal practice control group made significant gains but the tablet control group did not Masamu t (37) = 5.47, p <.001 Tablet Control t (34) = 1.86, p =.071, ns Normal Practice t (36) = 2.69, p =.011 38

(ii) Maths Curriculum Knowledge: 2 (Time) x 3 (Intervention group) mixed ANOVA Intervention group Maths Curriculum Knowledge n mean (sd) Pre Post Masamu Tablet 38 6.4 (6.3) 20.7 (10.3) Non- Masamu Tablet 35 8.3 (8.1) 15.1 (8.5) Normal Practice 37 5.5 (5.3) 10.8 (7.4) Results Significant main effect of time F (1, 107) = 147.24, p <.001 Significant main effect of group F (2, 107) = 6.12, p =.003 Masamu intervention group sig. better performance overall than normal practice control group Significant interaction between time and group F (2, 107) = 15.18, p <.001 Analysis of simple main effects Pre- test: no significant effect of group F (2, 107) = 1.70, p =.188 Post- test: significant effect of group F (2, 107) = 12.03, p <.001 Post- hoc tests with Bonferroni Tablet Control and Normal Practice, p =.115, ns Masamu sig > Tablet control, p =.024 Masamu sig > Normal Practice, p <.001 Masamu intervention group sig. better performance over time than both control groups All groups made significant improvements over time Masamu t (37) = 9.78, p <.001 Tablet Control t (34) = 5.57, p <.001 Normal Practice t (36) = 5.16, p <.001 (iii) Maths Curriculum Knowledge Generalisation: 2 (Time) x 3 (Intervention group) mixed ANOVA Intervention group Maths Curriculum Knowledge Generalisation n mean (sd) Post- Test Masamu Tablet 38 23.8 (9.1) Non- Masamu Tablet 31 17.4 (10.3) Normal Practice 32 20.7 (8.4) Results Significant main effect of group F (2, 101) = 4.12, p =.019 Post hoc tests with Bonferroni Tablet control and Normal Practice, p =.474, ns Masamu sig > Tablet control, p =.015 Masamu and Normal Practice, p =.497, ns Masamu intervention group sig. better performance over time than the tablet control group 39

Appendix III: Standard 3 ANOVA summary tables (i) Mathematical Concepts: 2 (Time) x 3 (Intervention group) mixed ANOVA Intervention group Mathematical Concepts n mean (sd) Pre Post Masamu Tablet 44 14.9 (5.7) 19.5 (5.2) Non- Masamu Tablet 44 15.4 (5.6) 19.3 (5.1) Normal Practice 43 14.9 (5.2) 18.6 (6.0) Results Significant main effect of time F (1, 117) = 56.91, p <.001 No significant main effect of group F (2, 117) =.19, p =.831, ns No significant interaction between time and group F (2, 117) =.24, p =.785, ns Over time significant improvement of S3 children but no differential effect of group (ii) Maths Curriculum Knowledge: 2 (Time) x 3 (Intervention group) mixed ANOVA Intervention group Maths Curriculum Knowledge n mean (sd) Pre Post Masamu Tablet 44 13.5 (9.3) 35.2 (7.0) Non- Masamu Tablet 44 14.8 (10.1) 24.9 (8.0) Normal Practice 43 11.00 (10.3) 23.4 (6.9) Results Significant main effect of time F (1, 128) = 351.92, p <.001 Significant main effect of group F (2, 128) = 10.29, p <.001 Masamu group sig. better performance overall than both control groups Significant interaction between time and group F (2, 128) = 20.45, p <.001 Analysis of simple main effects Pre- test: no significant effect of group F (2, 128) = 1.68, p =.191, ns Post- test: significant effect of group F (2, 128) = 34.25, p <.001 Post- hoc tests with Bonferroni Tablet Control and Normal Practice, p =.996, ns Masamu sig > Tablet Control, p <.001 Masamu sig > vs Normal Practice, p <.001 Masamu intervention group showed sig. better performance over time than both control groups All groups made significant improvements over time Masamu t (42) = 17.297, p <.001 Tablet Control t (37) = 6.849, p <.001 Normal Practice t (38) = 8.670, p <.001 40

(iii) Maths Curriculum Knowledge Generalisation: 2 (Time) x 3 (Intervention group) mixed ANOVA Intervention group Maths Curriculum Knowledge Generalisation mean (sd) n Post Masamu Tablet 43 33.5 (7.3) Non- Masamu Tablet 44 20.0 (6.7) Normal Practice 43 27.2 (8.1) Results Significant main effect of group F (2, 129) =.36.32, p <.001 Post hoc analyses with Bonferroni Tablet Control sig < Normal Practice p <.001 Masamu sig > Tablet Control p <.001 Masamu sig > Normal Practice p <.001 Masamu intervention group significantly better than both control groups on the maths curriculum knowledge generalisation paper and pencil test 41

Appendix IV: Gender ANOVA summary table Combined maths ability: 2 (Gender) x 2 (Time) x 2 (Intervention group) mixed ANOVA Intervention group Masamu ipad Learning Centre Normal Practice Classroom Standard Gender Combined maths ability/49 n mean (sd) Pre Post 1 F 10 2.3 (1.4) 7.4 (5.4) M 12 2.1 (1.9) 5.6 (3.8) 2 F 18 6.3 (5.2) 15.6 (8.5) M 20 8.5 (6.3) 19.5 (7.1) 3 F 22 14.5 (6.9) 28.3 (5.0) M 22 13.9 (6.7) 26.4 (5.9) 1 F 5 3.6 (3.0) 5.5 (4.7) M 15 2.9 (2.7) 5.0 (3.9) 2 F 17 6.0 (4.7) 9.0 (6.4) M 20 7.7 (5.4) 12.1 (6.4) 3 F 24 11.7 (4.4) 20.4 (5.9) M 19 14.6 (6.5) 21.8 (4.4) Results Signifcant main effect fo time F (1, 200) = 348.81, p <.001 Significant main effect of group F (1, 200) = 7.17, p =.008 No significant main effect of gender F (1, 200) =.07, p =.799 Significant interaction between time and group F (1, 200) = 34.52, p <.001 No significant interaction between time and gender F (1, 200) =.96, p =.329 No significant interaction between group and gender F (1, 200) =.02, p =.799 No significant interaction between time, group and gender F (1, 200) =.13, p =.720 No significant effects of gender across intervention groups and across time Note: No significant effects of gender were found when the two measures of mathematical ability (MC and CK) were examined separately. 42

Appendix V: Partial correlations between basic skills and mathematical abilities Partial correlations (r), controlling for age (months) between pre and post intervention performance on the tests of Mathematical Concepts (MC) and Maths Curriculum Knowledge (CK) and the neuropsychological tests of Manual Processing Speed (MPS), Manual Coordination (MCO), Visual Attention (VA), Short Term Memory (STM), Working Memory (WM) and Spatial Intelligence (SIQ) Tasks/Time Correlation coefficient r Whole matched sample Masamu ipad Learning Centre Intervention group Non- Masamu ipad Learning Centre Normal Practice Classroom MC df = 280 df = 101 df = 76 df = 97 CK Pre.50.72.55.18 Post.68.78.62.68 MC df = 280 df = 101 df = 76 df = 97 Speeded tasks Negative correlations expected MPS Pre -.21 -.27 -.36 -.13 Post -.15 (279) -.31 -.09 (75) -.07 MCO Pre -.11 (237) -.14 (79).09 -.20 (76) Post -.19 -.35 -.02 -.12 VA Pre -.23 -.24 -.05 -.27 Post -.35 -.44 -.39 -.20 Accuracy tasks Positive correlations expected STM Pre.20 (224).21 (74).38 (69).20 (75) Post.30 (279).29.15 (75).39 WM Pre.14 (248).02 (79).25.12 (87) Post -.10 (238) -.20 (79).01 -.16 (77) SIQ Pre.21 (248).15.29.18 Post.24 (238).18 (79).21.24 (77) CK df = 280 df = 101 df = 76 df = 97 Speeded tasks Negative correlations expected MPS Pre -.17 -.21 -.25 -.13 Post -.21 (279) -.32 -.28 (75) -.14 MCO Pre -.09 (237) -.14 (79).13 -.24 (76) Post -.22 -.33 -.09 -.13 VA Pre -.15 -.20 -.16 -.05 Post -.46 -.55 -.43 -.25 Accuracy tasks Positive correlations expected STM Pre.34 (224).27 (74).41 (69).36 (75) Post.34 (279).46.20 (75).37 WM Pre -.02 (248) -.13 (79).14 -.23 (87) Post.01 (238) -.09 (79).03 -.11 (77) SIQ Pre.38 (248).35.43.30 Post.34 (238).29 (79).26.29 (77) Significant Correlations (p <.05, at least, 2- tailed) highlighted in bold df when different to that reported in table is given in parenthesis after corresponding r value 43

Appendix VI: Pre and post intervention performance on the Ambitions and Attitudes Questionnaire (AAQ) for the matched sample of children Most frequent response (given as a percentage) provided for each question of the AAQ at pre- test and post- test for the matched sample receiving the Masamu Tablet Intervention group compared to the Normal Practice control group across Standard 1-3 Standard 1 2 3 AAQ question Intervention group Masamu Tablet Normal Practice Pre- test Post- test Pre- test Post- test Answer % Answer % Answer % Answer % Babies 5 50.0 5 40.0 Money 5 54.5 5 60.0 Job 4 45.5 4 45.0 Leave school 5 45.5 5 45.0 Home NA 5 54.5 NA 5 40.0 School 5 45.5 1 40.0 Gadgets 5 50.0 1 45.0 Friends 5 40.9 1 40.0 Hurt 5 40.9 5 45.0 Present 5 40.9 1 35.0 Babies 2 39.5 2 31.6 2 48.6 2 45.9 Money 5 50.0 5 81.6 5 54.1 5 83.8 Job 5 36.8 4 73.7 4 48.6 4 64.9 Leave school 5 57.9 5 78.9 5 54.1 5 73.0 Home 5 57.9 5 50.0 5 54.1 5 48.6 School 5 44.7 5 34.2 5 59.5 5 35.1 Gadgets 5 44.7 5 26.3 5 45.9 5 37.8 Friends 5 44.7 5 36.8 5 43.2 5 43.2 Hurt 5 44.7 5 31.6 5 56.8 5 40.5 Present 5 34.2 5 36.8 5 54.1 5 51.4 Babies 2 52.3 2 56.8 2 72.1 2 69.8 Money 5 65.9 5 88.6 5 81.4 5 74.4 Job 4 59.1 4 61.4 4 44.2 4 46.5 Leave school 5 77.3 5 84.1 5 62.8 5 81.4 Home 1 36.4 5 47.7 5 34.9 5 39.5 School 5 38.6 5 31.8 5 48.8 5 44.2 Gadgets 5 31.8 5 31.8 5 51.2 5 44.2 Friends 5 40.9 4 40.9 5 55.8 5 46.5 Hurt 3 31.8 3 40.9 3 34.9 5 41.9 Present 5 43.2 5 56.8 5 53.5 5 53.5 Control questions 9 & 10 are given in bold and italics and answers where the most frequent group response is in the opposite direction to that anticpated are highlighted 44