The Impact of an In-School CAL Program on Student Math Test Scores

Transcription

1 Integrating Computer Assisted Learning into a Regular Curriculum: Evidence from a Randomized Experiment in Rural Schools in Shaanxi Di Mo, Linxiu Zhang, Qinghe Qu, Matthew Boswell, Scott Rozelle Working Paper 247 December 2012 reapchina.org/reap.stanford.edu Introduction Developing and developed countries alike have taken the initiative to confront the long-standing challenge of delivering quality education to poor and disadvantaged 1

2 populations (e.g. Glewwe and Kremer, 2006; Planty et al., 2008; World Bank, 2004). Research has shown investment in traditional educational inputs alone has not been fully successful in promoting learning outcomes. Researchers have studied educational inputs such as textbooks and flipcharts (Glewwe and Moulin, 2002; Glewwe and Zitzewitz, 2004), teacher training (Jacob and Lefgren, 2004) and/or other monetary or in-kind educational inputs (e.g. Hanushek, 1986, 1995). None of these have been shown to consistently and effectively improve learning in developing or developed countries. Recently, attention has been placed on whether or not these trends hold true for alternative inputs, such as computer-assisted learning programs (CAL e.g., Banerjee et al. 2007; Barrow, 2008; Linden, 2008). Computer-assisted learning programs utilize computers and modern computing technologies (both software and hardware devices) to enhance learning through computerized instruction, drills and exercises (Kirkpatrick and Cuban, 1998; Present s Committee of Advisors on Science and Technology, 1997). With well-designed educational games, the program has been expected to sustain interest and curiosity of students, and lead to gains in student performance. So far, the empirical evidence on the impact of such programs has been mixed. Early studies in Israel and the United States found little consistent evidence on whether or not the application of computer technology in school instruction has beneficial effects for student academic achievement (e.g. Angrist and Lavy, 2002; Fuchs and Woosmann, 2004; Goolsbee and Guryan, 2006). Later studies evaluating specific CAL programs through randomized experiments, also have found mixed evidence. For example, both Dynarski et al. (2007) and Krueger and Rouse (2004) found no significant gain in math and reading test scores 2

3 from CAL programs for students in the United States. In contrast, Barrow, Markman and Rouse (2008) found a computer-assisted learning program improved student math test scores by 0.17 standard deviations in Chicago schools. Although relatively few in number, evaluations conducted in developing countries mostly show CAL has had positive effects on student test scores (Banerjee et al., 2007; He, Linden and MacLeod, 2008; Linden, 2008). An important limitation to these studies is that they usually examine CAL as an educational input, and do not consider whether or not the program is rolled out as an in-school program or out-of-school program. Miettinen (1999) defines in-school programs as those that occur during regular school hours and/or those that are organized as formal classroom activities. In contrast, out-of-school programs take place after school hours and/or are built around less formal group activities. Out-of-school programs can be run without the restrictions of the formal classroom (e.g., the limited scope of activities that are allowed to take place under the supervision of formal teachers), and without substituting for other learning activities in regular classes. Such programs, such as remedial education camps and other after school programs, have therefore been found to ultimately lead to higher levels of learning (Hull and Schultz, 2002). Out-of-school programs are also potentially easier to design in a way that can cater to the needs of individuals, as has been shown with professional training programs that teach workplace skills (Dias et al., 1999). However, the differences between the out-of-school and in-school programs may disappear if in-school programs are organized to use the educational resources efficiently (Cole, 1996). Despite the benefits, out-of-school 3

4 programs often require schools to make structural changes and/or sometimes are dependent on the extraordinary effort of teachers or volunteers. As a consequence, it is often suggested that for programs to be sustainable, they need to ultimately be incorporated into the regular school-day curriculum (Underwood et al., 2000). Unfortunately, despite the large possible differences between in-school and out-of-schools programs, there exists little evidence comparing the differential effects. This is also true of in-school and out-of school CAL programs. Previous studies have suggested that the lack of an impact of a CAL program on educational performance in developed countries may be attributable to the substitution away from the well-equipped classrooms and relatively high quality of instruction that students enjoy during normal school hours (e.g. Banerjee et al., 2007). To our knowledge, there is only one study actually addressing such differences for CAL interventions. Linden (2008) found an in-school CAL program in India to be ineffective. In contrast, the same research team showed that an out-of-school version of CAL did improve student learning. According to Linden (2008), one of the reasons for the difference in impact is that the in-school program appears to have created a substitution of effort/time away from other productive, in-class activities (such as formal instruction by the teacher in math and language). The out-of-school program instead appears to have served as a complement to existing resources. Despite the interesting findings, the Linden study did not take into account how the in-school CAL program was incorporated into the regular school curriculum. India is known for short school days. For example, schools often provide only five to six hours 4

5 per day of instruction (reported in the study by Duflo et al., 2008). There is also high absenteeism of teachers during regular school hours (Chaudhury, et al., 2005) which may have contributed to the ineffectiveness of the in-school CAL program, particularly if the supervising teachers frequently missed the CAL sessions. As mentioned above, the effectiveness of an in-school program depends on which resources it is replacing. If CAL is used as substitute for relatively unproductive periods of time during the school day (e.g., if CAL was run during an open or free period, or during a period that was not being fully utilized such as an art or music class in a school that does not have resources for such classes), the effect of CAL on learning may be greater since there might be less of an offsetting substitution effect. Does CAL work in China? Previous studies that employed large-scale randomized controlled trials to evaluate the effectiveness of CAL on student learning have shown that CAL can significantly improve the math and Chinese test scores of rural students in China (Lai et al., 2011; Mo et al., 2012; Lai et al., 2012a; Lai et al., 2012b). However, the initial studies were all designed as after-school supplementary activities. In other words, the CAL interventions were all run as out-of-school programs. As such, in these previous efforts to test CAL, there was no danger of substituting for other activities run by the teachers during the regular teaching day. The question remains whether an in-school CAL program in Chinese schools will be equally effective in improving learning when it is implemented during regular school hours. These questions are of particular relevance to China. In order to narrow the digital divide between rural and urban schools, the government has recently invested in 5

6 improving the computing infrastructure of rural public schools (Yuan, 2012). By 2011, 86 percent of the rural public schools had set up computer rooms with an average of 17 computers in each school (Yang et al., 2012). China s Ministry of Education, however, has even more ambitious plans. The recently announced 12 th Five-Year Plan for Integrating Information Technology into Education aspires to set up a computer room in every rural school by 2020 (Ministry of Education, 2012). Since the plan requires such an enormous investment of fiscal resources, it is important to learn how to effectively use the new computing resources. Unfortunately, China s rural schools face many constraints in providing quality computer-based education. Teachers at rural schools typically do not have the qualifications or materials necessary to promote learning in computer classes (Lai et al., 2011). Teachers lack the training and/or motivation to adequately instruct students and pique the interest of students in using computers for learning. There is also a shortage of curricula to use during computing class, particularly in poor rural areas (Yang et al., 2012). Although 69 percent of students in rural public schools have computer classes, research shows that few schools have employed computers and/or educational software for instructional purposes in core academic subjects. Even when they do, computer classes are frequently cancelled due to a lack of teachers and instructional materials. If CAL classes could occur in periods already assigned for computer classes which are not being used effectively it may be that in-school CAL classes in China could bring the proven benefits of the CAL program without any offsetting effects. In other words, if CAL could be conducted during the regularly scheduled, but, poorly 6

7 utilized computer class period, CAL in China may be made to be both integrative and supplemental. The overall goal of this paper is to explore the impact of an in-school CAL program on the academic outcomes of an underserved student population in a developing country. To achieve this goal, we pursue three specific objectives. First, we examine the impact of an in-school CAL math program on student academic performance in math (measured by standardized test scores). Second, we compare the effects of the in-school program to those of an out-of-school program (previously reported in Lai et al., 2011). Third, we seek to understand whether the in-school CAL program had any serious substitution effects that made the program less effective than it would have been had it been run as an out-of-school CAL program. The rest of the paper is organized as follows. The next section reviews the study by Lai et al. (2011) which reports on the results of an out-of-school CAL program. The third section describes the current study s methodology, including the research design and sampling, intervention design, data collection and statistical approach. The remaining sections present the results from the study, discuss the findings and conclude. Out-of-school CAL Program in Shaanxi Province In our first attempt to implement the CAL program as a purely supplemental (out-of-school) activity, we conducted a cluster-rct in Shaanxi Province during the school year. A total of 5943 students in 72 schools of Shaanxi rural schools were involved in the study. Among the students, 2726 students were boarding students and the 7

8 other 3074 students were non-boarders. The boarding students constituted the sample for the study; the non-boarders only were used as additional controls. In other words, only the boarding students in 36 treatment schools received the CAL program. To test of the effectiveness of CAL, we designed an intervention that was implemented in the 36 treatment schools. In its most basic form (complete details of the CAL intervention are included in the next section), the CAL program consisted of a remedial, game-based CAL program in math that was held outside of regular school hours among boarding students. To test the effectiveness of the CAL programs, students in both treatment and control schools were given standardized math tests before the start of the program and at the end of its implementation. According to the analysis in Lai et al. (2012a), which is replicated in Figure 1 and Table 1 in this paper, the CAL program had a positive and significant effect on the math test scores of students in the treatment schools. Overall, as seen in Figure 1, scores went up by 0.12 standard deviations. Table 1 includes the results of the regressions and shows (when using a full model--see section below for more details) that the impact on third grade students was 0.18 standard deviations (column 6) and the fifth grade students was 0.07 standard deviations (column 7). Although the result for fifth grade students is not significant, we found significant impact on the students up to the 70th percentile in post-cal math score. In other words, the CAL program had a significant impact on the worst performing fifth-grade students. Although the results in Lai et al. (2012a) were positive, significant and robust, the design of the intervention means that the external validity of the findings was somewhat 8

9 limited. The program was only implemented among boarding school students after school. As this was primarily an efficacy trial, sessions were held after school on purpose in order to minimize any substitution effects that might have offset the impact of the program. We put the intervention after school and included only boarding school students who otherwise would not have been engaged in any other academic activities during the period when the CAL program was offered to the students. It is unlikely that if the entire school system were to adopt this program that it would be restricted to such a small subset of students. Instead, it is likely that school officials would seek to implement the CAL program during the course of the normal school day, so as to benefit all students (both boarding and those that live at home). That is why we designed the current study to integrate CAL into the course of the normal school day schedule and expand participants to all students in the sample grades. Sampling, Data and Methods Sampling and the Process of Randomization We conducted a clustered (at the school level) RCT of Computer-Assisted Learning (CAL) in Shaanxi rural schools during the academic school year. A total of 5267 students in 72 schools of Shaanxi rural schools were involved in the study. The study covered third grade and fifth grade students. Choosing the sample consisted of several steps. First, to focus our study on students from poor rural areas, we restricted our sample frame to four counties randomly selected out of the ten counties in Ankang Prefecture, the prefecture that covers one of the poorest areas in the southern region of Shaanxi Province. Shaanxi Province is a large (it 9

10 has a population of nearly 40 million), rural (more than 60 percent of the population live in rural areas) and poor province in northwestern China. The average per capita income of these four counties is only around 4000 RMB (around $600) per year in 2011, which is far below rural China s average per capita income of 6977 RMB in the same year (CNBS, 2011). After choosing the counties, we obtained a comprehensive list of all wanxiao (those elementary schools with six full grades grade one through grade six) in each of the four counties from the Department of Education of Ankang Prefecture. We included all 72 schools that met the above criterion in our sample. Within the sample schools, we included both third grade and fifth grade students in the 72 schools in our sample. We chose third grade and fifth grade students for two reasons. First, at the time of the launch of the project, we only had remedial tutoring material for student from the third grade to sixth grade. It is for this reason that we did not choose students from first grade or second grade. Second, a subset of the fourth grade and sixth grade students in the school had already participated in a pilot project during the previous academic year. In order to avoid confounding the treatment effect, we chose to focus the intervention on third grade and fifth grade students. Again, none of these students had ever participated in a CAL program prior to the academic year. So who was included in the sample? In fact, all of the third grade and fifth grade students in the 72 sample schools were included. In a previous study we had only included students who boarded at school. In this study we included students who were boarding at school and students that were living at home. Of the total number of students 10

11 involved in the study (5267), 2279 were third grade students and 2988 were fifth grade students (Figure 2). Although at the time of the baseline survey the main sample included a total of 72 schools and 5267 students, for various reasons (mainly because of school transfers and extended absences due to illness or injuries), there was some attrition by the end of the study. By the time of the evaluation survey we were able to follow up with 4757 students in the 72 sample schools (Figure 2, final row). In other words, 4757 out of the initial 5267 students (who took the baseline survey) were included in our evaluation survey and were part of the subsequent statistical analysis; 9.8 percent of the sample dropped out between the baseline and endline surveys. There were 249 attrited students (10.9 percent) from the third grade and 261 attrited students (8.7 percent) from the fifth grade. Fortunately for the study s integrity, there were no variables that were systematically related between the characteristics of students and their attrition status (Table 2). After choosing the 72 schools for our sample, we randomly chose 36 schools from these 72 schools to receive the CAL intervention. As the CAL intervention engaged both of third grade and fifth grade students, the 2435 students of the third and fifth grades in the 36 treatment schools constitute the treatment group (Figure 2). Among these students, there were 1067 third grade students and 1368 fifth grade students. The 2832 students of the same grade (1212 from the third grade and 1620 from the fifth grade) in the other 36 schools served as the control group. Due to the attrition, there were 4757 students left in our final analytic sample, among whom 2220 were in the 36 treatment schools, and 2537 were in the control schools. 11

12 We used a set of student characteristics to check the validity of the random assignment. According to our data, we found that for all student characteristics, none of the differences between the treatment and control groups were statistically significant (Table 3, columns 1 and 2). In addition, the differences are almost all small in magnitude. Experiment Arm/Intervention The main intervention involved computer-assisted math remedial tutoring sessions which were designed to incorporate the regular in-class math curriculum for the entire school year Under the monitoring of two teacher-supervisors trained by our research group, the students in the treatment group had two 40-minute CAL sessions per week as regular classes in school. The sessions were mandatory and attendance was taken by the teacher-supervisors. According to our protocol, the CAL sessions were supposed to be given during the normal computer class time period. We chose the computer class time periods in China since typically these are reserved for teaching non-academic material. The content (instructional videos and games) of each session were designed for improving students basic competencies in the uniform national math curriculum and they were exactly the same for all students in each of the treatment groups. During each session, two students shared one computer and played math games designed to help students review and practice the basic math material that was being taught in their regular school math classes. In a typical session, the students first watched an animated video that reviewed the material that they were receiving instruction on during their regular math class sessions in that week. The students then played math games with animated 12

13 characters to practice the skills introduced in the video lecture. i If a student had a math-related question, he/she was encouraged to discuss with his/her teammate (the one he/she shared the computer with). The students were not allowed to discuss with other teams or the teacher-supervisor. Our protocol required that the teachers could only help students with scheduling, computer hardware issues and software operations. In fact, according to our observations, the sessions were so intense that the students were almost always exclusively focused on their computers. There was little communication among the groups or between any of the groups and the teacher-supervisor. The CAL software had enough content and exercise games to cover the math course materials for the entire school year and the material was sufficient to provide 80 minutes of remedial tutoring per week (two 40-minute sessions). The intervention team spent considerable time to prepare the necessary hardware, software, CAL curriculum and program implementation protocol in a way that would both facilitate smooth implementation of the CAL program and avoid confounding influences that might bias our results. As the first step, to meet the hardware requirements of the CAL program, we acquired (by way of donation from Dell, Inc.) 640 brand new identical desktop computers and installed CAL software package on these desktops. We then removed all pre-installed software that would not be used during the CAL intervention (such as Windows built-in games and Microsoft Office) and disabled the Internet and USB functions on all of the computers. In this way, not only could we prevent students or teachers from using the program computers for other purposes that might affect the operation of the regular CAL program but we could also avoid the interruptions that 13

14 might otherwise be caused by accidental deletion of the CAL software or the introduction of viruses. Sealing the computers also ensured the quality of our evaluation of the program effects without capturing any other confounding influences (spillovers) if students gain knowledge by having access to other sources of information such as Internet. It also prevented teachers/students from the control schools from copying our CAL software onto their computers. With both software and hardware ready, we then worked out a detailed CAL curriculum and implementation protocol. The protocol was targeted exclusively at the teacher-supervisors that were responsible for implementing the CAL program in each school. The CAL curriculum was designed to keep in pace with progress of school instruction on a week by week basis. This was done so that our CAL sessions provided timely review and practice of the knowledge and skills that were introduced and covered as part of their regular math class. One of the most important jobs of the teacher-supervisor was to make sure the weekly CAL sessions were proceeded on a pace that matched the pace in the students regular math classes. Because this work was clearly beyond the scope of their normal classroom duties, we compensated the teacher-supervisors with a monthly stipend of 100 yuan (approximately 15 USD), an amount roughly equivalent to 15 percent of the wage of a typical rural teacher. All teacher-supervisors of the 36 treatment schools also participated in a two-day mandatory training program. CAL Control Group (the students in the 36 control schools) 14

15 The third-grade and fifth-grade students in the 36 control schools constituted the CAL control group. Students in the control group did not receive any CAL intervention. To avoid any types of the spillover effects of the CAL intervention, the principals, teachers and students (and their parents) of the control schools were not informed of the CAL project. The research team did not visit the control schools except for during the baseline and final evaluation surveys. The students in the control group took their regular math and computer classes at school as usual. Data Collection The research group conducted two rounds of surveys in the 72 control and treatment schools. The first-round survey was a baseline survey conducted with all third and fifth graders in the 72 schools in June 2011 at the end of the spring semester and before any implementation of CAL program had begun. The second-round survey was a final evaluation survey conducted at the end of the program in June In each round of the survey, the enumeration team visited each school (treatment and control schools) and conducted a two-block survey. In the first block students were given a standardized math test, which gave us our main outcome variable. The math test included questions (tests in different grades and rounds included slightly different numbers of questions). All the questions were chosen to not repeat the questions that were contained in the exercises in the CAL software. Students were required to finish tests in each subject in 25 minutes. Time limits were strictly enforced. In the second block enumerators collected data on the characteristics of students and their families. From this part of the survey we are able to create demographic and 15

16 socioeconomic variables. The dataset includes measures of each student s gender, age (measured in years), whether the student is a boarding student, have the student ever repeated grade, if the student is the only child of his or her family, the age of father, and the age of mother (both measured in years), father s education level (father has at least junior high school degree), mother s education level (mother has at least junior high school degree) and parental care (at least one parent lives at home). To create the indicator of family wealth, we documented the ownership of household appliances to proxy the family asset value. The variable of family wealth equals 1 if the family assets are higher than the median value and 0 otherwise. Statistical Methods We used both unadjusted and adjusted ordinary least squares (OLS) regression analyses to estimate how the academic outcome changed in the treatment group relative to the control group. Our unadjusted analysis regressed changes in the outcome variable (i.e. post-program math test score minus pre-program math test score) on a dummy variable of the treatment (CAL intervention) status. We used adjusted analysis as well to improve statistical efficiency. In all regressions, we accounted for the clustered nature of our sample by constructing Huber-White standard errors corrected for class-level clustering (relaxing the assumption that disturbance terms are independent and identically distributed within classes). The unadjusted model is:!y isc =! + " "treatment s +# isc, (1) where!y isc is the change in the outcome variable during the program period for child i in school s and class c, treatment s is a dummy variable for a student attending a 16

17 treatment school (equals to one for students in the treatment group and zero otherwise) and! isc is a random disturbance term clustered at the class level. By construction, the coefficient of the dummy variable treatment s, β is equal to the unconditional difference in the change in the outcome (!y isc ) between the treatment and control groups over the program period. In other words, β measures how the treatment group changed in the standardized math test score levels during the program period relative to the control group. In order to improve the efficiency of the estimation, we built on the unadjusted model in equation (1) by including a set of control variables:!y isc =! + " "treatment s +# " y 0isc + X isc $ +% isc (2) where all the variables and parameters are the same as those in equation (1), except that we added a set of control variables. Specifically, we controlled for y 0isc, the pre-program math test score and Chinese test score for student i in school s and class c, and X isc, a vector of additional control variables. The variables in X isc are student and family characteristics (gender, age, boarding student, ever repeated grade, only child, age of father, age of mother, father has at least junior high school degree, mother has at least junior high school degree, at least one parent lives at home and family wealth). By including y 0isc and X isc as control variables, β in equation (2) provides an unbiased, efficient estimate of the CAL treatment effect. Results According to our analysis, it appears that the CAL program significantly improved the academic performance of the third grade students in the sample treatment schools 17

18 (Figure 4). The third grade pre-test standardized test scores are slightly lower (though insignificantly so) in the control group than in the treatment group (Panel A). ii After the CAL intervention, the treatment group improved more in math than did the control group (Panel A). The difference in change in standardized math test scores between the two groups is 0.15 standard deviations for the third grade students (Panel B). The size of the CAL program effect also is comparable to the findings in other CAL evaluations outside of China that observed beneficial effects of CAL on student performance (e.g., Barrow, 2008; Banerjee et al., 2007; Linden, 2008). The results are similar for the fifth grade students (Figure 5); the data show that students in the fifth grade also improved significantly more in their math test than the students in the control group. The fifth grade pre-test scores are slightly lower in the control group than in the treatment group though not in a statistically significant way (Panel A). After the CAL intervention, students in the treatment group improved more in math than did those in the control group. The difference in the change in standardized math test scores between the two groups is 0.13 standard deviations for the fifth graders (Panel B). The multivariate regression analyses (adjusted and unadjusted) are consistent with our graphical descriptive analysis (Table 4). Using only third grade students or only fifth grade students, the estimated CAL treatment effects on math test scores are equal to 0.15 standard deviations for the third grade students in the unadjusted model (Table 4, column 1, row 1), and 0.13 standard deviations for the fifth grade students (Table 4, column 2, row 1). Unsurprisingly, given the nature of the unadjusted model, these estimates are 18

19 close to the effect sizes in the descriptive statistics in Figures 4 and 5 (discussed above). Both of the coefficients of interest in the unadjusted model are not significant (Table 4, column 1 & 2, row 1). However, when we add the control variables (using the adjusted model Equation 2), the more efficient estimates show that the CAL program had a positive and significant impact on the standardized math scores of the students. The estimated treatment effect for third grade students increases to 0.18 standard deviations (Table 4, column 3, row 1) and is significant at the 5% level. The estimated treatment effect for fifth grade students also increases to 0.14 standard deviations and is significant at the 10% level (Table 4, row 1 column 4). Using Equation (2) the results largely show that the effect of the CAL treatment does not systematically vary by boarding status (Table 5). iii When we divide the sample into two parts (one sub-sample consisting of only the boarding school students; and the other sub-sample consisting of only the non boarding school students), the point estimate of the impact of the CAL treatment on the standardized math scores for both third and fifth grade boarding and non-boarding school students are positive. The scores of the grade three boarding students improved by 0.15 standard deviations (row 1, column 3); the scores of the grade three non-boarding school students increased by 0.25 standard deviations (column 4). Likewise, the scores of both grade five boarding and non boarding school students increased by 0.19 and 0.12 standard deviations. Although the precision of the estimates for the grade three and grade five students differed for boarding and non-boarding school students (non-boarding was significant for grade three students; 19

20 boarding was significant for grade five students), when testing for differences between boarding and non-boarding students, in the case of both grade three and grade five students, there is no statistically significant difference (with a p-value of for grade three students; p-value of for grade five students). In summary, then, our results demonstrate that the impact of the in-school program on either the boarding or non-boarding school students does not vary significantly from that of the out-of-school CAL program reported by Lai et al. (2012a): both programs have improved the performance of most of the students that participated in the CAL treatment programs by non-trivial magnitudes (between 0.1 and 0.25 standard deviations). Specifically, by integrating the CAL program into the course of the regular schooling day, the program improved boarding student performance of the third grade students by 0.15 standard deviations, which was only slightly lower than the treatment effect of 0.18 standard deviations that was found in the out-of-school CAL program. For the fifth grade boarding students, the out-of-school CAL program had an impact of 0.11 standard deviations (for students up to the 70 th percentile using their pretest scores), while the in-school program had a impact of 0.19 standard deviations. Both programs have proven to be a success by consistently improving student performance by more than one-tenth of a standard deviation. There also were similar estimated effect sizes (in some cases higher) for non-boarding school students in both the third and fifth grade. Substitution Effect of the In-school CAL Program Despite finding more or less similar results for the out-of-school and in-school CAL treatments, it is still important to further explore whether or not there were any 20

21 substitution effects. This is important to understand if the government decided to upscale this to other schools outside of the sample area. If we can identify substitution effects in the particular environment of the current trial, it will be useful to judge what the overall effect of CAL might be in circumstances in which a different environment would (ex-ante) likely to have scope for larger (e.g., in schools in which all class periods were already being used efficiently) or smaller (e.g., in schools in which some class periods were not being used at all or less efficiently) substitution effects. So how can we examine if there was a substitution effect? One imperfection in the way that the CAL program was implemented by the schools during the course of the trial actually provides an opportunity to study whether there was indeed any substitution effect. During the evaluation survey we discovered that, although all the treatment schools implemented the CAL program for the requisite two classes a week (as required by the protocol), a subset of the schools on their own volition did not fully integrate both of the CAL sessions into the regular school day program. Instead, some schools (17% of all the treatment schools) decided to hold part of the CAL intervention after school. In 16% of the schools, the principal arranged to have all of the CAL classes occur out of regular school hours. Hence, in the rest of this section, by exploiting such school-to-school differences in implementation, we investigate whether the impact of the CAL program would differ if it were organized as a pure in-school program, as a pure out-of-school program, or as a mix of the two (part in-school; part out-of-school). One concern that we need to address when doing this analysis in seeking to identify the substitution effect is that it is possible (likely) that there is selection bias involved. In other words, it may very well be that the schools that arranged the CAL classes differently did so not randomly, but for some reason that might be related to the overall impact of CAL. For example, through qualitative interviews performed by the 21

22 authors after the implementation of CAL had been completed, the principals of the schools that did not fully integrate CAL into the curriculum gave us several reasons for having moved the sessions to after school hours. First, schools in which almost all students boarded at school were more likely to implement CAL after school. This decision, according to the principal, was made for two reasons. Most obviously, both students and a number of the teachers were still at school after the regular school hours. Hence, organizing the CAL sessions after school was relatively easy. In addition, when schools host a majority of boarding school students, they often are located in relatively remote areas. Because teachers (that did not stay at school each night) often lived far away from school, the length of the school days was often shorter (meaning that frequently the school did not have many free class periods for non-core courses that is for courses besides math, Chinese and English). Second, our interviews with the principals of larger and better equipped schools discovered that these schools were somewhat more inclined to make the CAL program a fully integrated, in-school program. Such schools, according to the findings of the interviews, were better at enforcing the school calendar that had been set up for the CAL treatment during the early part of the program. Such schools often were able to do this because they have larger and better equipped computer rooms. The school days were also somewhat longer. Larger schools also had more stable faculties. In smaller schools there were more incidences when a teacher-cum-cal coordinator transferred out of the school (or quit) in the middle of the school year. When this happened, another teacher needed to be assigned to take over the teacher s classes (including the CAL class) until a new teacher was hired. This often caused an interruption in the CAL program and, in several cases, schools arranged the CAL sessions to continue but after school. 22

23 The strategy we employ to study the effect of substitution is to estimate the change in math test scores on interactions between treatment and a measure of program integration (measured by variables that indicate if a school s CAL program was implemented as full integration as originally designed, partial integration or fully supplemental/out-of-school). We also control for other school-level characteristics to correct for school selection (to as great of degree as possible). The final estimation model is:!!!"# =! +!!"#$!%#&!! +!! (!!!!"#$!%#&!! ) +!! (!!!!"#$!%#&!! ) +!!!!! +!!!!! +!!!!"# +!!"#! +!!! +!!"# (3) which includes variables indicating the treatment status (!"#$!%#&!! ), dummy variables for full integration of CAL program (!!! ) and partial integration of CAL program (!!! ), and the interaction terms between treatment status and program integration. The excluded program integration dummy variable is for no integration of the CAL program (that is, having the CAL program entirely after school). Note this means that the base group includes the schools which implemented CAL after school. The model also includes all of the control variables in equation (2),!!!"# and!!"#. We have, in addition, included an additional set of school-level variables,!!, to capture the characteristics of boarding facilities, the school s size, the nature of the school s resources and some measures of teacher quality. Specifically, we include variables that measure the total number of boarding students; percentage of students that board; the area of the school (in square meters); whether the school had a playground (1=yes; 0=no); whether the school had a library (1=yes; 0=no); whether there was incentive pay for teachers; how much of the teacher s annual salary was due to incentive (in RMB); the number of teachers in the school; and whether most computers in the school were usable/functioning (1=yes; 0=no). 23

24 According to the results from the analysis using equation (3), although there are measureable substitution effects, students in both in-school (integrated) and out-of-school (supplemental) CAL programs benefited. In the case of both third grade and fifth grade students, Table 6 confirms that students significantly benefited from the in-school CAL program (columns 1&2, rows 1&2). The results also demonstrate, however, that the students were likely to benefit even more if the CAL program was organized after school (columns 1&2, row 1).The coefficients on the treatment variable, which indicates the treatment effect on the base group (the ones that took the CAL classes after school), are positive and significant at 5% level for both grades. It is even more striking to see, however, that the students who took CAL after school improved almost twice as much as the students that took the classes during school. Third grade students that were in the fully supplemental (out-of-school) CAL programs improved their math scores by 0.36 standard deviations (relative to those of the control group); fifth grade students improved by 0.26 standard deviations (row 1, columns 1 and 2). There is also some evidence suggesting that program integration may have reduced the CAL impact. The coefficients on the interaction terms between treatment and the variables that measure full integration and partial integration are both negative for both the third grade and the fifth grade students (rows 2 and 3, columns 1 and 2). These results suggest that full/partial integration makes less of an impact on student performance than fully implementing CAL after school. Although the coefficients are not significant on the interaction terms, this may be due to a lack of power (since the program was not fully powered to distinguish between these different types of schools). Consistent with our previous results, however, in-school CAL programs still have positive effects on the math scores of students as program integration does not fully offset the impact of CAL. In this version of the regressions, the estimated treatment effect of the in-school CAL programs 24

25 is 0.27 standard deviations for the third grade students and 0.20 standard deviations for the fifth grade students. Overall, then, our findings in this section of the paper seem to suggest that there are moderate substitution effects, but, the effects are not enough to fully offset the impact of the CAL program. Of course, selection bias may be driving these results (and there may be no substitution effect at all). In other CAL papers in China (e.g., Lai et al., 2011; Lai et al., 2012a), it was found that the students with poorer performance (before the CAL program) were more positively impacted by CAL. In our findings, it is possible that this is driving the differences in measured impacts of CAL between the fully supplemental schools and the integrated schools. As seen from our interviews, fully supplemental schools tended to be smaller and more remote schools with more boarding school students. Such schools have been shown by others (e.g., Luo et al., 2009; Shi et al., 2012) to have poorer average levels of educational performance, in general. Hence, taken together (supplemental schools may be poorer performing; and poor performing sets of students enjoy larger CAL impacts), the greater impact of CAL in the supplemental schools may be due to the differences in the quality of the students, not from substitution. Several factors, however, suggest that this may not be the case in our current study (which would lend more support for the existence of substitution effects). First, when we compare the school characteristics between the fully integrated schools and the fully supplemental schools, we find that the differences in these characteristics are not significantly different (Table 7, columns 1-7). Second, when we look at the differences in performance (using baseline standardized math test scores) among fully integrated and fully supplemental schools we find the fully supplemental schools do have lower student math test scores than the fully integrated ones (Table7, row 8). However, when running heterogeneous effects in our model between students with poor performance at the 25

26 baseline and those with better performance at the baseline, we find no statistically significant difference in impact (Table 8, columns 1&2, row 2). Moreover, when running the regression using equation (3), we have controlled for baseline math scores and many other variables that may affect student learning (see Table 2 for the control variables). Therefore, these findings suggest that, in fact, it may be substitution effects (and not selection effects due to different quality of students in the schools that have different degrees of integration) that are leading to the observed pattern of the impact of CAL among fully integrated, partially integrated and fully supplemental CAL program schools. Conclusion In this paper we present the results from a randomized field experiment of a Computer Assisted Learning (CAL) program in 72 rural public schools in Ankang, Shaanxi. The study involves around 5267 third-grade and fifth-grade students. To evaluate the effectiveness of the program we randomly chose 36 schools from the entire sample as treatment schools and the third and fifth grade students in these schools received the CAL intervention. The remaining 36 schools served as control schools. The main intervention was designed to be a math CAL program held during regular school hours (during a regularly scheduled computer class). The students were offered 40 minutes of shared computer time after school, twice a week. During the sessions students played computer-based games that required them to practice using their knowledge of math and relatively simple problem solving skills. The CAL program was tailored to the regular school math curriculum and was remedial in nature, providing the students with drills and exercises with different levels of difficulty. 26

27 Our results indicate that an in-school CAL program has significantly improved student academic outcomes. Two 40-minute CAL math sessions per week increased the student standardized math scores by 0.18 standard deviations for the third grade students and 0.14 standard deviations for the fifth grade students. The gains in this program do not vary much from the out-of-school pilot program reported by Lai et al. (2012a). In this way, we believe our current paper adds more support for the policy proposition that the new effort to put computer rooms into all schools in poor rural areas of Western China should be accompanied by efforts to make sure that effective educational-based software (like CAL) are also available (and that teachers are trained well in using them). Although the program has proven to be a success, we find some evidence that there are ways to make the program even more successful. The results of our study suggest that if the CAL programs were organized as out-of-school, supplemental programs, the impact would have been larger. In other words, we have evidence that supports the conclusion that there are substitution effects when CAL programs are run as integrated, in-school programs. The result also shows that the substitution effect of an in-school program (on standardized test scores) is around 0.1 standard deviation (a moderately large effect). However, it needs to be stressed that more research and caution is needed before all CAL programs are to be run as out-of-school programs. Although there may be larger effects for out-of-school programs (and assuming these effects are real substitution effects and not due to selection bias as discussed above), it may be that the gains from doing so would have to come at the cost of excluding some groups of students. Running an 27

28 after-school program means that some students (such as the non-boarding students that have long daily commutes to school) would be less likely to participate. This means that schools need to balance the greater effectiveness of out-of-school programs with the greater inclusiveness of in-school programs. Of course, as we also have seen in the paper, the final decision on out-of-school versus in-school may be different for different schools. The programs in the case of this trial indeed did work in all schools. Perhaps if schools were given the flexibility and the proper training they could decide what degree of integration would give them the best mix of effectiveness and equity. 28

29 References Angrist, J., and Lavy, V New Evidence on Classroom Computers and Pupil Learning. The Economic Journal, 112 (October): Banerjee, A., Cole, S., Duflo, E., and Linden, L Remedying Education: Evidence from Two Randomized Experiments in India. Quarterly Journal of Economics. 122 (3): Barrow, L., Markman, L., and Rouse, C Technology s Edge: The Educational Benefits of Computer-Aided Instruction. National Bureau of Economic Research working paper No Chaudhury, N., Hammer, J., Kremer, M., Muralidharan, and Rogers, K. F. H Teacher Absence in India: A Snapshot. Journal of the European Economic Association, 3(2): Cole, M Cultural psychology: A once and future discipline. Cambridge, MA: Harvard University Press. Dias, P., Freedman, A., Medway, P., and Paré, A Worlds apart: Acting and writing in academic and workplace contexts. Mahwah, NJ: Lawrence Erlbaum Duflo, E., Hanna, R. and Ryan, S.P Incentives Work: Getting Teachers to Come to School. Mimeo, MIT Economics Department. Dynarski, M Effectiveness of Reading and Mathematics Software Products: Findings from the 1st Student Cohort. Mathematica Report. Fuchs, T., and Woosmann, L Computers and Student Learning: Bivariate and Multivariate Evidence on the Availability and Use of Computers at Home and at Schools. Brussels Economic Review, 47: Glewwe, P., and Kremer, M Schools, Teachers, and Education Outcomes in Developing Countries. Handbook on the Economics of Education. (New York, NY: Elsevier), 2: Glewwe, P., and Moulin, S Textbooks and Test Scores: Evidence from a Prospective Evaluation in Kenya. BREAD Working Paper, Cambridge, MA. Glewwe, P., and Zitzewitz, E Retrospective vs. Prospective Analyses of School Inputs: The Case of Flip Charts in Kenya. Journal of Development Economics, 74: Goolsbee, A., and Guryan, J The Impact of Internet Subsidies in Public Schools. The Review of Economics and Statistics, MIT Press, 88 (2): Hanushek, E The Economics of Schooling: Production and Efficiency in Public Schools. Journal of Economic Literature, 24: Hanushek, E Interpreting Recent Research on Schooling in Developing Countries. World Bank Research Observer, 10: He, F., Linden, L., and MacLeod, M How to Teach English in India: Testing the Relative Productivity of Instruction Methods within the Pratham English Language Education Program. Working Paper. Columbia University Department of Economics. Hull, G., and Schultz, K. (Eds.) School s out! Bridging out-of-school literacies with classroom practice. New York: Teachers College Press. Jacob, B., and Lefgren, L The Impact of Teacher Training on Student 29