Financial Education for Children and Youth: A Systematic Review and Meta-analysis

Transcription

1 Financial Education for Children and Youth: A Systematic Review and Meta-analysis Aflatoun Working Paper C Support: This review was made possible through the generous support of Aflatoun Child Social and Financial Education. Suggested Citation: O Prey L and Shephard D (2014) Financial Education for Children and Youth: A Systematic Review and Meta-analysis. Aflatoun Working Paper C. Accessed at Corresponding Author: Llorenc O Prey, School for Policy Studies, University of Bristol, 8 Priory Road, Bristol, BS8 1TZ, United Kingdom: llorenc.oprey@bristol.ac.uk Working Paper C 1

2 Executive Summary This paper reports on the findings of the systematic review and meta-analysis exploring the effectiveness of financial education aimed at children and youth. Overall, 21 independent experimental studies were uncovered, suggesting that the evidence base is broader than previously thought. The studies reported on a broad range of programmes, including the content that was taught, the way it was delivered, and the context in which children and youth experienced their education. There exists a varying degree of methodological quality across included studies, with some of the most rigorous studies being the most recent suggesting that the evidence base is improving over time. Where there was sufficient and comparable data to do so, the outcomes reported in each study were combined to produce an overall effect-size. The results suggest that financial education is effective in improving knowledge, attitudes and behaviours. The strongest and most significant finding was regarding knowledge gains. The meta-analysis found overall modest improvements in attitudinal and behaviour change. This confirms that generally, financial education does produce tangible gains in financial capability amongst children and youth. Summary of Overall Effects of Financial Education: Knowledge gains: 0.18 (0.09, 0.27) Attitudinal change: 0.08 (0.01, 0.15) Behaviour change: 0.07 (0.03, 0.11) Standard Mean Differences (95% Confidence Intervals) Random Effects Model Compared to research exploring other educational programmes, this metaanalysis presents robust findings in support of financial education. Across the studies included in this review, there was variation in the effectiveness of different programmes, with some providing greater effects than others. The paper goes onto explore and expand on these issues, and their potential implications for practice. Working Paper C 2

3 Background This review collects and synthesizes all the available experimental evidence that analyses the change brought about by financial education programmes aimed at children and youth. Through meta-analysis the review will explore the links between education programmes and intended outcomes. The Systematic Review seeks to: 1. Explore the efficacy and effectiveness of financial education programmes aimed at children and youth. 2. Explore causal links between financial education and outcomes. 3. Within different approaches and programmes, identify which types of programmes are showing promise and which are less effective. 4. Provide a definitive statement of the quality and scope of the evidence regarding financial education aimed at children and youth. Search of the Literature The authors of the review completed a systematic search of the literature, which produced over 1400 potential studies. These studies were then screened against a number of criteria, including: Intervention: Financial Education Population: Children and Youth Study Design: Randomized Control Trail Design Studies that met all of the inclusion criteria were included in the review. The screening process found 21 unique studies reported in 15 papers. Two papers were found to report on 8 independent studies. A summary of the literature search can be found overleaf. Working Paper C 3

4 Table, summary of the literature search: Ongoing studies that could not be included: The search also uncovered a number of studies that are ongoing, and were unable to be included in the review because they were still under completion. Tajikistan CRCT of the efficacy of a programme called Aflateen+ that combines a youth social and financial programme with a sexual and reproductive health intervention designed for young women at risk of early marriage. Data collection is complete with the final analysis expected in Peru CRCT of two variations of the youth social and financial education programme called Aflateen being led by Matthew Bird. The study seeks to determine if a different goal setting mechanism in the curriculum affects savings behavior. Data collection is complete with the final analysis expected in Colombia RCT looking into the effectiveness of youth access to accounts combined with text messaging once or twice a month, with one group having financial education oriented text messaging instead of just reminders to save. China CRCT looking into the effectiveness of the Aflatoun and Aflateen Working Paper C 4

5 social and financial education curriculum by comparing a control group that receives no programme, with the standard programme group plus individual savings, and a third group using the same curriculum but combining it with a group savings mechanism. Rwanda CRCT of a social and financial education for children called Aflatoun. The study is using both children outcomes and teacher observation methods to determine causal pathways between teacher methodology and children s social and financial outcomes. A CRCT of a school-based financial education programme in Turkey, conducted by Alan Ertaç. A cluster randomized control trial evaluating the YouthSave Ghana Experiment, with results expected later in 2014 (Chowa et al, 2012). Suubi Bridges a follow-up experiment looking into the Suubi project in Uganda. Led by Fred Ssewamala. Characteristics of Included Studies The 21 unique studies display a broad range of characteristics in the way they were designed and executed. Methodology Nine studies reported clustering at the classroom or school level, and 11 at the individual level, and one used an encouragement design analysed at the individual level. Most studies employed a treatment as usual control, in which participants continue as normal (f = 14). One study reported using a wait-list control, in which participants assigned to the control have financial education delayed until after the study (Supanantaroek, 2013). Three studies used multiple trial arms to explore the effectiveness of different treatments. Most studies sought to create a balance sample size between intervention and control. Most studies used pre- and post- testing to explore change over time. Some studies reported collecting data at multiple time-points, including after the programme had finished. Participants Studies were located across the globe, although most were conducted in the US (f = 11). Uganda (f = 5), Ghana (f = 1), Tunisia (f = 1), Tanzania (f = 1), Brazil (f = 1), and Italy (f = 1) were also represented. Studies varied in size, the smallest involving just 67 young people, and the largest initially involving over 240,000. All studies sought to understand co-educational programmes that were delivered to both boys and girls without exception. Intervention All studies provided at least partial descriptions of the financial education components under investigation. 11 unique programmes were investigated, Working Paper C 5

6 including two trials that explored the effectiveness of Aflatoun programme in Ghana and Uganda, three trials looking into the effectiveness of the SEED/Suubi Project in Uganda, and six trials of variations of Visa s online credit education program. Across the different programmes, there was considerable diversity in the content and mode of delivery. The majority were classroom based, in which children and youth would access content through exercises and materials delivered by a teacher (f = 12). Other studies explored media delivered content, including the web based materials (f = 6), computer based materials linked to classroom content (f = 2), and one explored the effectiveness of edutainment, a TV show which included a financial education component. All studies included some form of financial education that looked to develop either knowledge and skills, or the importance of saving and asset accumulation. For an outline of study characteristics, please see overleaf. Working Paper C 6

7 Table: Summary of Included Studies: Study 1 Becchetti et al (2012) 2 Berry et al (2013) 3 Bruhn et al (2013) 4 Faircloth et al (1986) 5 Gartner et al (2005a) 6 Gartner et al (2005b) Type of Mode of Country Focus Duration Trial Delivery N CRCT Italy Financial 16 hours Classroom 944 education CRCT Ghana Mixed, 20 hours Classroom 5,000 Financial and Social Education CRCT Brazil Financial 72 hours Classroom 26,000 education CRCT USA Financial education RCT USA Financial education RCT USA Financial education 3 hours Classroom, on computer At own Media, pace web At own pace Media, web ,112 80,982 7 Gartner et al (2005c) 8 Gartner et al (2005d) 9 Gartner et al (2005e) 10 Gartner et al (2005f) 11 Hinojosa et al (2009) 12 Jamison et al (2012) RCT USA Financial education RCT USA Financial education RCT USA Financial education RCT USA Financial education RCT USA Financial education CRCT Uganda Financial education At own pace Media, web At own Media, pace web At own Media, pace web At own Media, pace mail and web 15 hours Classroom, on computer 82,600 8,190 11,686 39, Classes 15 hours Classroom 2, Jorgensen and Tonsburg (2011) Encouragement CRCT Tanzania Financial education 14 Karimli (2013) CRCT Uganda Mixed, Financial and Health Education 15 McLean (2010a) RCT USA Financial education 14 hours Media, TV 2, to 20 hours Classroom hours Classroom McLean (2010b) 17 McNeil et al (2008) 18 Premand et al (2012) 19 Ssewamala et al (2008) 20 Ssewamala et al (2009) 21 Supanantaroek et al (2012) RCT USA Financial education RCT USA Financial education RCT Tunisia Mixed, Financial and Entrepreneurial Education CRCT Uganda Mixed, Financial and Health Education CRCT Uganda Mixed, Financial and Health Education CRCT Uganda Mixed, Financial and Social 12 hours Classroom 75 2 hours Classroom weeks intensive Classroom 1, hours Classroom 96 Not stated Classroom hours Classroom 1,746 Working Paper C 7

8 Risk of Bias in Included Studies Before analysing and synthesizing the results of each study, it is important to understand how each study was completed, and if any potential weaknesses in its design or execution could have influenced (biased) the results. A bias is a systematic error in results or inferences. Biases can operate in either direction: different biases can lead to an underestimation or overestimation of the true intervention effect. Cochrane Handbook (2014) Against each study, assessments were made across a number of potential risks by the review authors using the Cochrane Collaboration s risk of bias tool (Higgens et al, 2011). With each potential risk, reviewers made an independent assessment either a low risk of bias, one that was unlikely to significantly alter the results, or a high risk of bias, one that seriously weakens confidence in the results. In cases where there was insufficient information from a paper to make an accurate assessment, attempts were made to seek clarification from authors. Several authors did not respond, in which case the judgment was left as an unclear risk of bias: a plausible bias that raises some doubt about the results. In completing the risk of bias assessment, a number of methodological weaknesses were identified. Random sequence generation: All studies reported randomly assigning participants to either intervention or control groups. Eight studies provided sufficient information to ascertain that the methods used to randomly allocate schools or participants were at a low risk of bias. A number of studies (f = 3) reported using stratification as a means of ensuring the comparability of intervention and control groups. This process involved dividing schools or participants into groups according to certain characteristics, such as geographical location (eg, urban and rural) prior to random allocation. The precise method of randomisation was not reported in 10 studies, and we were unable to clarify with authors. This makes it difficult to ascertain if selection bias was likely to have occurred and, if so, in which direction it may have biased the results. Therefore an assessment of unclear risk of bias was made. Allocation Concealment: Allocation concealment was not practically possible in the majority of studies, especially where allocation occurred at the school or district level. Where adequate random sequence generation methods were used, a low risk of bias has been assumed, as it would ensure that allocation was down to chance. Working Paper C 8

9 Bruhn et al (2013) reported that after allocation, some school districts requested that at least one school be allocated into the intervention school, requiring three schools, chosen at random, to be transferred over from control to intervention group. In this instance a low risk of bias has been assumed as steps were taken to ensure a measure of chance in the final allocation. Of the remaining studies, it was not possible to determine whether allocation concealment had been appropriately carried out, so therefore an unclear risk of bias was assumed. Blinding of participants and personnel: In many instances, it was not feasible to completely blind participants or personnel as to which arm the school or individual had been allocated, especially for those in clusters or in the intervention arm. In this instance, a low risk of bias has been assumed. Hinojosa et al (2009) reported that allocation influenced attrition, in which teachers that found themselves in the control group either requested to be in the intervention group or decided not to continue with the study. In this instance a high risk of bias has been assumed as it was clear that an absence of blinding significantly influenced data collection. Blinding of outcome assessment: Three studies reported that it was not possible to blind data collectors as to the assignment of particular schools or participants, however took steps to control for sampling or measurement error. In this instance a low risk of bias has been assumed. It was not clear from 18 studies that steps were taken to either blind data collectors as to the allocation of intervention and control arms, or to take measures to control for reporting bias. In this instance, an unclear risk of bias is assumed as it is not possible to rule out if and how this bias may have influenced results. Incomplete data: Most studies reported high rates of attrition (drop-out). The way in which missing data from attrition is handled can have a significant impact on the results. Two studies reported handling attrition and missing data by analysing outcomes based on intention-to-treat analysis. This form of analysis attempts to control for missing data, and therefore assumes a low risk of bias. Similarly, in studies that had low or no reported attrition (f= 4) a low risk of bias is assumed. In cases (f = 3) where it was unclear how attrition was handled, and therefore the potential to bias the results, an unclear risk of bias was assumed. In one case (Jamison et al 2014), the amount of attrition did not differ between groups; Working Paper C 9

10 however, a mixture of analytical methods to deal with missing data made the existence and direction of any bias difficult to interpret as such an unclear risk of bias was assumed. Two studies appeared to report a treatment-on-treated analysis of the results. This ignores drop-out, and only analyses data for those participants that completed the study. This type of analysis is likely to overestimate the effectiveness of a programme to the extent to which those that dropped out display different characteristics and outcomes to those that finished the programme, for example, lower levels of motivation or knowledge gains. In this instance a high risk of bias has been assumed. All of the Gartner et al (2005) studies reported extremely high rates of attrition. This was due to the nature of the intervention, however it does present significant potential to bias the final results. Therefore a high risk of bias has been assumed. For Becchetti et al (2013), it was not clear exactly how sample size and attrition were reflected in the final analysis, with different analyses and papers reporting different sample sizes. Attempts to contact the authors to clarify were unsuccessful, so in this instance a high risk of bias is assumed. The effect of Becchetti on the Financial Knowledge meta-analysis was subjected to sensitivity analysis but did not substantively affect the overall findings. Selective reporting: We were unable to find published protocols for any included study. This made it difficult to determine selective reporting bias when a study only publishes data that favours a particular hypothesis. All studies reported outcomes that are consistent with the study objectives, and where there was no reason to suggest selective reporting such as not reporting negative findings, a low risk of bias is assumed. Where there was insufficient information, an unclear risk of bias was assumed. With Berry et al (2013), it was unclear if the composite scores used to communicate the results accurately reflected all the data that was collected. Without clarification, a high risk of bias has been assumed. Other sources of bias: Included studies were also screened for a variety of other potential risks. Most studies reported using survey questionnaires (instruments) to collect data that were designed specifically for the programme under investigation. It was not always clear if assessments were made to ensure that instruments were both valid and reliable. Validity, the extent to which an instrument produces stable and consistent results, and reliability, the extent to which it accurately measures the concepts, skills and competencies under investigation, can influence data collection in a number of ways. Without a process of validation assessing whether an instrument is both accurate (valid) and consistent (reliable) it is difficult to determine how the results may have been affected. Another aspect that was assessed for potential bias was implementation fidelity Working Paper C 10

11 the degree to which a programme was delivered as intended. Through exploring and measuring whether an intervention has been implemented with fidelity, it is possible to understand how and why a programme works, and the extent to which outcomes could be improved. It raises confidence in the results, especially in those studies that report no effect. When fidelity is not assessed, it is difficult to determine whether the observed results are due to the intervention itself, or an undocumented counterfactual. Very few studies reported taking steps to ensure implementation fidelity. Most reported the programmed number of hours children and youth were expected to attend, but did not go onto explore the actual number of hours they received. This is perhaps the most basic measure of implementation fidelity in education research, and included studies were marked as being potentially open to bias if, at a minimum, participant exposure was not reported to have been observed. A summary of risks can be viewed in this graph, and a table overleaf. Table: Risk of Bias Graph Working Paper C 11

12 Working Paper C 12

13 Publication Bias: Another potential bias is that of publication bias, the possibility that publication of research depends on the nature and direction of the results. For example, some areas of effectiveness research tend to only make studies that report positive outcomes towards an intervention publically available, over those that do not. Assuming the assumption holds that meta-analysed data displays a normative distribution, then we are able to assess symmetry in the data. Though subjective assessment of funnel plots, it appears that data is broadly symmetrical for knowledge and attitudes, suggesting that there is a low risk of publication bias in the sample of included studies. However, it is notable that all studies of behaviour were on the positive end. Table: Funnel Plot of Knowledge Outcomes: Table: Funnel Plot of Attitude Outcomes: Working Paper C 13

14 Table: Funnel Plot of Behaviour Outcomes: Working Paper C 14

15 Results Of the included studies, we extracted all outcome data that could be located using published and unpublished sources, including contacting authors. It was not possible to include data from some studies as there was insufficient information to fully synthesis the data within a meta-analysis. The Gartner et al studies (2005) and Ssewamala (2008) did not provide sufficient data to include their analyses, and were contacted to provide clarification without success. In the case of Ssewamala, discussions with members of the research group indicated that the SEED study did not collect financial outcomes for both treatment and control groups because the study was focused on psychosocial outcomes only. Knowledge Eleven studies provided comparable knowledge outcomes. In comparing the results between the intervention and control groups after the programme had finished, six studies found clear knowledge gains for the intervention group. Five studies produced inconclusive results, in which the 95% confidence interval crosses the line of no effect. Table: Knowledge Outcomes, Random Effects Model: Combining the results of all studies together using a random effects model, we find that financial education has a moderate effect on knowledge outcomes (0.18). Statistical heterogeneity is high and significant (I 2 = 88%, p<0.001), suggesting that the results should be interpreted with caution, and that effect sizes reported may be accounted for by the differences within the included studies. Note: The highest effect size of 0.95 (Faircloth et al, 1986) tested knowledge immediately after the intervention which is likely to inflate the outcome in favour of intervention. The study was comparing computer assisted financial education with reading based financial education. Working Paper C 15

16 Table: Knowledge Outcomes, Fixed Effects Model: Using a fixed effects model to combine the results, we find similar positive results (0.18, CI 0.16, 0.20). Again, statistical heterogeneity is high and significant (I 2 = 88%, p<0.001). Working Paper C 16

17 Attitudes Nine studies provided comparable attitudinal outcomes. We found that three studies produced clear results in favour of the financial education, and six studies produced inconclusive results. Table: Attitude outcomes, Random Effects Model: In combining the results together, the intervention improves attitudinal outcomes by 0.08 standard deviations in favour of financial education. The effect size is small with the 95% confidence interval approaching the line of no effect (0.01 to 0.15). Heterogeneity was moderate and statistically significant (I 2 = 79%, p < 0.001). Table: Attitude outcomes, Fixed Effects Model: Using a fixed effects model to combine the results, we find a small effect size (0.06, CI 0.04, 0.09). Note: The data reported here for Ssewamala et al (2009) has been treated slightly differently. Due to significant variation in baseline differences between groups, gain scores were used to establish an overall effect size. This gives a more accurate indication of the overall change brought about by the programme. Working Paper C 17

18 Behaviours Six studies provided comparable behaviour outcomes, of which five produced results in favour of the financial education. One study, Premand et al, reported whether participants accessed loans as a result of attending the intervention. The other four studies assessed self-reported savings behaviour. Table: Behaviour outcomes, Random Effects Model: When all six studies results were combined applying a Random Effects model the effect-size was 0.07 (95% CI = 0.03, 0.11, p < 0.001). Statistical heterogeneity was relatively low and statistically insignificant (I 2 = 53%, p = 0.06), suggesting that the results are more consistent than knowledge and attitude findings. Table: Behaviour outcomes, Fixed Effects Model: The use of a Fixed Effects model reduced the overall effect-size to 0.06 with small variance (95% CI: 0.04, 0.07). This is due the fact that the one study, Bruhn et al (2013), dominated the results in this model with 81.7% of the weight, indicating that the Random Effects model is preferable. Note: Karimli (2013) data was transformed from an Odds Ratio to SMD to include into the meta-analysis using the method suggested in Chinn However, sensitivity analysis indicated that the inclusion of this data did not substantively change the overall findings due to the wide confidence interval and low weighting of the study. Working Paper C 18

19 Sub-group Analyses A number of sub-group analyses were conducted using a random effects model to explore trends within the data. Study Design: The use of an active control had the expected effect on the knowledge outcomes. Studies with an active control (f = 3) showed no statistically significant treatment effect (Z = 0.78, p = 0.43) while studies with a control group not receiving any form of financial education showed a modest but robust treatment effect of 0.18 standard deviations. The same is true for attitudinal outcomes, where studies with active controls (f = 2) showed no overall effect (Z = 1.42, p = 0.16) while studies with an inactive or TAU control showed an overall treatment effect of 0.12 standard deviations. Risk of bias found that attrition bias was the most significant risk within included studies. Studies with high risk of attrition bias were compared to those with inconclusive and low risk studies, with mixed results. High-risk studies reported slightly higher treatment effect and a broader confidence interval (SMD 0.19, CI 0.03, 0.36). Studies with a high risk of bias that exploring attitudinal change appear to have overestimated effects (0.30 compared to the overall effect of 0.06 in low risk studies). There was insufficient data to explore the effect of study design on behaviours, as all included studies shared the same characteristics around type of control and low risk of attrition bias. Table: Summary of study design sub group analyses (random effects): Knowledge Attitudes Type of Control Incomplete Outcome Data Inactive Ctrl Active Ctrl High RoB Other RoB No. of measures Effect Point Estimate CI (Lower, 0.09, , , , 0.26 Upper) Z (p) 3.99 (0.0001) 0.78 (0.43) 2.32 (0.02) 1.73 (0.08) I 2 90% 85% 87% 84% No. of measures Effect Point Estimate CI (Lower, 0.02, , , , 0.11 Upper) Z (p) 2.50 (0.01) 1.42 (0.16) 0.90 (0.37) 2.34 (0.02) I 2 82% 0% 96% 51% Working Paper C 19

20 Intervention: There was considerable variation in the composition and the mode of delivery of content across studies. Those programmes delivering just financial education were compared with those that delivered financial education plus another component, including social, health or entrepreneurship education. The results suggest that for knowledge outcomes, those programmes that delivered just financial education were more effective (financial only: 0.19, financial plus: 0.16), although the results should be interpreted with caution as the observations of financial plus cross the line of no effect. However, for both attitudinal and behavioural outcomes, the results suggested that financial plus produced larger effects compared to financial only. For attitudinal outcomes, studies of financial plus programmes found a treatment effect of 0.17 standard deviations while those of financial only found no effect. For behavioural outcomes, financial plus programmes had an effect of 0.13 standard deviations while the single study of financial only had a smaller effect of Comparing the number of hours of tuition children and youth receive within programmes suggested that those delivering less than 40 hours of tuition produced greater knowledge, attitudinal and behaviour change than those delivering more than 40 hours. Observations for attitudes were not significant however. In a further sub-group analysis, financial education delivered in the classroom was compared to content delivered through media, such as TV, computers or the internet. The results found higher knowledge gains for media driven interventions (0.47, compared to 0.14 for classroom based programmes), although two of the media studies took place in a classroom (Faircloth et al 1986) and in one case included complementary in class content (Hinojosa et al 2009). There is also unpublished data to suggest that the Jorgensen and Tonsburg (2011) study produced unintended outcomes, as those in the intervention group reported lower attendance at school and lower test scores at follow-up. There was insufficient data to explore attitudes and behaviour outcomes. Table on following page: Working Paper C 20

21 Table: Summary of intervention sub group analyses (random effects): Knowledg e Attitudes Behaviour Content Length of Programme Mode of Delivery Financial Plus Financial Only <40 Hours >40 Hours Class Media No. of measures Effect Point Estimate CI (Lower, -0.07, , , , , , 0.89 Upper) Z (p) 1.36 (0.17) ( 0.17) 3, (0.0004) (0.06) (0.002) (0.03) I 2 93% 85% 85% 93% 89% 68% No. of NA NA measures Effect Point Estimate CI (Lower, 0.03, , , -0.01, 0.12 Upper) 0.27 Z (p) 2.34 (0.02) 0.11 (0.91) (0.11) (0.19) I 2 85% 91% 80% 87% No. of NA NA measures Effect Point Estimate CI (Lower, 0.02, , , , 0.07 Upper) Z (p) 2.40 (0.02) 7.15 (<0.0001) 3.07 (0.002) 7.3 (0.0001) I 2 71% 0 38% 0% Working Paper C 21

22 Population Similarly, there was significant variation in the context and age range of participants found within studies. In exploring the role of context in shaping outcomes, the influence of human development was analysed. Drawing on the Human Development Index (HDI, UNDP, 2013), studies that took place in countries not rated as having very high HDI, were compared with those that were. The results suggest that programmes operating in countries with very high HDI produced similar knowledge gains, however, were significantly poorer at changing attitudes. There was insufficient data to explore behaviour change and country HDI. By separating those studies aimed at children (0-14) and youth (15+), sub-group analysis found greater impact across knowledge attitudes and behaviours for programmes aimed at younger children. With attitudinal outcomes, however, the effect size crosses the line of no effect. Table: Summary of population sub group analyses (random effects): Knowledge Attitudes Behaviour Context Age range Very High HDI Other HDI Children (0-14) Youth (15+) No. of measures Effect Point Estimate CI (Lower, -0.17, , , , 0.26 Upper) Z (p) 1.03 (0.30) 4.42 (0.001) 2.21 (0.03) 2.73 (0.006) I No. of measures Effect Point Estimate CI (Lower, -0.57, , , , 0.09 Upper) Z (p) 1.42 (0.16) 2.5 (0.01) 1.84 (0.07) 0.78 (0.44) I % 72% No. of NA NA 3 3 measures Effect Point Estimate CI (Lower, 0.09, , 0.07 Upper) Z (p) 3.89 (<0.0001) 7.39 (<0.0001) I 2 7% 0% Working Paper C 22

23 Conclusions Understanding the Results: Summary of Overall Effects of Financial Education: Knowledge gains: Modest robust positive effect 0.18 (0.09, 0.27) Attitudinal change: Small positive effect 0.08 (0.01, 0.15) Behaviour change: Small robust positive effect 0.07 (0.03, 0.11) Standard Mean Differences (95% Confidence Intervals) Random Effects Model The meta-analysis produced conclusive results suggesting financial education is effective in developing financial capabilities. Knowledge change represents a modest positive result, with small gains in both attitude and behaviour outcomes. Overall this indicates that financial education can be effective across a range of outcome domains and that the evidence base is much broader than is often supposed. The heterogeneity among the studies suggests that different interventions, combinations, contexts and targeting strategies will produce different results. Therefore, in order to assist in informing practice, the authors identified the strongest interventions in each outcome category. The strongest programmes for financial knowledge change were The Stock Market Game (Hinojosa et al 2009) and the Brazil high school programme (Bruhn et al 2013). Both of these interventions focused only on financial topics, included an in class component, and arguably included participants with stronger numeracy skills than some of the other study contexts. The best programme for changing financial attitudes was the Suubi Project that combined financial topics, health topics, and financial access with young single and double orphans in Uganda (Karimli 2013; Ssewamala et al 2009). The combined effect of these two studies on financial attitudes was 0.45 (0.03, 0.87). This combined intervention included younger participants (age 13), addressed health and psychosocial issues, and provided a 200% matched savings account to incentivise asset accumulation. Across studies, those that targeted younger populations had a stronger effect on all domains, although for attitudes this did not meet the standard cut-off for statistical significance. The strongest programme for changing financial behaviour was the Aflatoun curriculum which combines social and financial topics and targets younger populations (Berry et al 2013; Supanantoroek 2013). The combined effect of these two studies was 0.16 (0.08, 0.24), twice that of the overall effect. Across interventions, those that combined financial education with another component, targeted young beneficiaries and had a shorter overall intervention were more effective at changing behaviour. Working Paper C 23

24 Comparing the Results with other Educational Research: There is no universal guideline or rule of thumb for judging the practical importance or substantive significance of a standardized effect-size estimate for an intervention, especially when comparing them to other educational programmes. Within education, the US Department of Education defines an effect size of.25 of a standard deviation as a substantially important effect (2013, p27). This can be misleading however, as a particular effect-size may present a relatively small or large impact depending on the benchmark that is most relevant. For example, 0.1 may be considered to be a relatively small gain in reading test scores over an academic year, but may be considered a large effect if demonstrating improved school attendance over the same period. A meta-analysis of 181 studies exploring curriculum or broad instructional and free-standing programmes in the US found an average standard deviation of 0.32 (Lipsey et al, 2012). These included a range of programmes, such as reading, maths or science curricula, and multisession programmes in a general subject area. The figure should also be viewed with caution, however, as it draws on unstandardised data exploring a broad range of outcomes. It focuses on studies in the US, and does not control for other potential confounding variables including the amount of time children spend receiving tuition. In other educational research, a meta-analysis of over 800 meta-analyses found that on average, teachers in front of the class produced an effect of.24 of a standard deviation, and that programmed instruction on average produced an effect of.18 (Hattie, 2009). In comparison with other curriculum programmes and the US Dept. for Education s benchmark the overall effect of financial education observed in this review and meta-analysis are substantial for financial knowledge outcomes and modest for attitude and behaviour outcomes. However, it should be noted that most educational interventions measure outcomes with a focus on knowledge and skills rather than attitudes and behaviour, so we postulate that the smaller effect sizes found in this review may be of comparable size to attitude and behaviour shifts in other education programmes. Furthermore, the educational focus of the contexts should be taken into account. For children and youth themselves, the financial education programmes that they have been given access to represent a small and arguably peripheral part of their overall education. Gains in knowledge, attitudes and behaviours across financial educational programmes featured in this review therefore may be proportional to larger, and more intensive educational programmes. Implications for Research: The existing research base exploring the effectiveness of financial education is of varying methodological quality. Future research should seek to improve the overall quality of the literature. The application of robust experimental methods, the minimisation of potential bias, especially around allocation bias, should be a priority. The reporting of trials should also be improved, especially with regards Working Paper C 24

25 to methodology, implementation fidelity, and data analysis. This will help improve the comparability, transparency and utility of the research base. Although there exists a significant research base, the review has exposed significant gaps in our understanding. The paucity of experimental research exploring financial education in Europe and Asia should cause concern. There is a debate currently around the importance of numeracy in financial educational knowledge outcomes. Non-experimental studies have suggested that when you control for numeracy, financial education has no effect. Meanwhile, both Hinojosa et al (2009) and Bruhn et al (2013) found that financial education had improved overall math scores within the intervention group. This suggests that there is a bidirectional relationship, and that financial education may contribute to numeracy development while numeracy can be important in shaping successful financial education outcomes. The role of numeracy should be further investigated within experimental studies, in order to better understand that relationship. Working Paper C 25

26 Annex A: Summary of Included Studies Becchetti et al (2012) Published: Intervention: Target Group: N and Attrition: Location: Type of Study: Data collection: Primary outcomes: Results: Meta-analysis: Summary: Becchetti L, Caiazza S and Coviello D (2012) Financial Education and Investment Attitudes in High Schools in Italy Classroom-based financial education. Female and male students in their last year of high school (unclear, large variations among analyses) Schools across Rome, Milan, and Genova, Italy. Clustered RCT with treatment and control. Self-administered survey Financial literacy, including knowledge of financial concepts, savings preferences and investment attitudes. Impact on improving knowledge and attitudes, especially amongst those with the lowest financial literacy prior to programme. Knowledge The study explored how financial education affects financial literacy and investment attitudes amongst a sample of high school students in Italy. The treatment group received 16 hours of tuition relating to financial literacy. The study found that the treatment group displayed increased levels of financial literacy and the propensity to read and understand economic articles and principles, although the results are not statistically significant. The treatment group also recorded reduced risky behaviors, however they did not increase prudent behaviors. A sub-group analysis found differences based on location, starting financial literacy, and other characteristics. The analysis also uncovered baseline differences indicating imperfect randomization. Working Paper C 26

27 Berry et al (2013) Published: Berry J, Karlan D and Predhan M (2013) Evaluating the Efficacy of School Based Financial Education Programs in Ghana Intervention: Classroom-based financial education. Target Group: Female and male pupils aged between 9 and 14. N and Attrition: 5363 (75) Location: Schools in semi-urban and rural Ghana, including Greater Accra, Volta and in the Western District. Type of Study: Clustered RCT with two treatments, Aflatoun and Honest Money Box interventions, and a control group. Data collection: Primary outcomes: Results: Meta-analysis: Self-administered survey Financial well-being of students and their families, cognitive function, and perspectives on savings and time and risk preference. Students in HMB-assigned schools showed positive and significant increases in their overall attitudes toward savings, compared to students in control schools. However, students in both Aflatoun and HMB schools showed positive impacts on savings behavior compared to students in control schools: those students were more likely to save any money, and those students were also much more likely to save money at school. The pooled treatment effects are used in this metaanalysis. Knowledge, Attitudes and Behaviours Working Paper C 27

28 Bruhn et al (2013) Published: Intervention: Target Group: N and Attrition: Location: Type of Study: Data collection: Primary outcomes: Results: Meta-analysis: Summary: Bruhn M, Bilal Zia, Arianna Legovini, Rogelio Marchelli (2013) Financial Literacy for High School Students and Their Parents: Evidence from Brazil Classroom-based financial education. High school students and their parents. 17,680 (Attrition varied across time-points) Brazil Cluster randomised control trial. Surveys to individual pupils and at the household level Budgeting, savings, and general financial management. Significant impact on improving knowledge, attitudes and behaviours. Knowledge, Attitudes and Behaviors The study assessed the impact of high school financial education in Brazil. It included nearly 900 schools students. Administration of the program through schools allowed for a broad coverage of content in the curriculum. Separate training was provided to a group of parents of the students to examine whether inside-the-household interactions influenced behavior. Results found that the program increased student financial knowledge by 0.20 of a standard deviation, which led to a 1.4 percentage point increase in savings behaviours - a relatively large and economically relevant effect. Both current attitudes and forward-looking intentions to save improved. A complementary workshop for parents induced children to save even more. Working Paper C 28

29 Faircloth A (1986) Published: Faircloth A (1986) Effectiveness of Computer-Assisted Instruction for Teaching Consumer Credit in the USA. Intervention: Computer assisted instruction, vrs supervised reading for teaching consumer credit. Target Group: High School students in grades 10 through 12 N and Attrition: 68 Location: USA, large rural high school. Type of Study: Randomised Control Trial, with each arm testing either computer assisted instruction or supervised reading control group. Data collection: Pre and post tests, and administrative data. Primary Students IQ, grade point average, attitude towards computers outcomes: and familiarity with computers. Results: Computer assisted instuction is an effective intervention for teaching consumer credit with diverse types of high school students. Meta-analysis: N/A Summary: Using a random sample of 68 high school students learning about consumer credit in home economics classes, the study measured learning of an experimental group using computerassisted instruction and a control group using supervised reading methods. Pretests and posttests indicated greater learning gains in the experimental group. Working Paper C 29

30 Gartner et al (2005a) Published: Intervention: Target Group: Gartner K and Todd R (2005) Effectiveness of Online Early Intervention Financial Education for Credit Cardholders in the USA Online, credit-card education programme. New credit card holders or card holders reaching the point of first delinquency. N and Attrition: 241,112 Location: Across the USA Type of Study: RCT Data collection: Administrative data on customer behaviour, repayment performance, delinquency etc. Primary Behaviour change amongst credit card holders. outcomes: Results: Completion of online credit education correlates with more responsible credit card usage, however the experiments don t prove that the education causes this behavior. Meta-analysis: Summary: N/A The first of six independent RCTs, this was conducted by the Bank Wells Fargo. It tested whether offering online credit card education to credit cardholders is effective in changing behavior. The targeted populations were either new cardholders or cardholders reaching the point of first delinquency. The experimental groups received brief counseling plus a referral to the VISA online credit education web site MoneyChoices.com. Experiments with college student cardholders found much more responsible behavior by those who choose to complete the online education. The other studies, however, found smaller overall differences between the control and experimental groups. These results were not statistically significant. Working Paper C 30

31 Gartner et al (2005b) Published: Intervention: Target Group: N and Attrition: Location: Type of Study: Data collection: Primary outcomes: Results: Meta-analysis: Summary: Gartner K and Todd R (2005) Effectiveness of Online Early Intervention Financial Education for Credit Cardholders in the USA Online, credit-card education programme. New credit card holders or card holders reaching the point of first delinquency. 80,982 with significant attrition. Across the USA RCT Administrative data on customer behaviour, repayment performance, delinquency etc. Behaviour change amongst credit card holders. Completion of online credit education correlates with more responsible credit card usage, however the experiments don t prove that the education causes this behavior. N/A Conducted by Target. Working Paper C 31

32 Gartner et al (2005c) Published: Intervention: Target Group: Gartner K and Todd R (2005) Effectiveness of Online Early Intervention Financial Education for Credit Cardholders in the USA Online, credit-card education programme. New credit card holders or card holders reaching the point of first delinquency. N and Attrition: 82,600 Location: Across the USA Type of Study: RCT Data collection: Administrative data on customer behaviour, repayment performance, delinquency etc. Primary Behaviour change amongst credit card holders. outcomes: Results: Completion of online credit education correlates with more responsible credit card usage, however the experiments don t prove that the education causes this behavior. Meta-analysis: Summary: N/A Conducted by US Bank, aimed at cardholders past the point of delinquency on credit card payments. Working Paper C 32

33 Gartner et al (2005d) Published: Intervention: Target Group: Gartner K and Todd R (2005) Effectiveness of Online Early Intervention Financial Education for Credit Cardholders in the USA Online, credit-card education programme. New credit card holders or card holders reaching the point of first delinquency. N and Attrition: 8190 Location: Across the USA Type of Study: RCT Data collection: Administrative data on customer behaviour, repayment performance, delinquency etc. Primary Behaviour change amongst credit card holders. outcomes: Results: Completion of online credit education correlates with more responsible credit card usage, however the experiments don t prove that the education causes this behavior. Meta-analysis: Summary: N/A Completed by US Bank, the study aimed at understanding intervention aimed at non-delinquent but high risk college students. Working Paper C 33

34 Gartner et al (2005e) Published: Intervention: Target Group: N and Attrition: Location: Type of Study: Data collection: Primary outcomes: Results: Meta-analysis: Summary: Gartner K and Todd R (2005) Effectiveness of Online Early Intervention Financial Education for Credit Cardholders in the USA Online, credit-card education programme. New credit card holders or card holders reaching the point of first delinquency. 11,686 with significant attrition. Across the USA RCT Administrative data on customer behaviour, repayment performance, delinquency etc. Behaviour change amongst credit card holders. Completion of online credit education correlates with more responsible credit card usage, however the experiments don t prove that the education causes this behavior. N/A Completed by US Bank, the study aimed at understanding intervention aimed at non-delinquent and low risk college students. Working Paper C 34

35 Gartner et al (2005f) Published: Intervention: Target Group: Gartner K and Todd R (2005) Effectiveness of Online Early Intervention Financial Education for Credit Cardholders in the USA Online, credit-card education programme. New credit card holders or card holders reaching the point of first delinquency. N and Attrition: 241,112, with an average age of 18 Location: Across the USA Type of Study: Three independent RCT s each exploring support offered to credit card holders. Data collection: Administrative data on customer behaviour, repayment performance, delinquency etc. Primary Behaviour change amongst credit card holders. outcomes: Results: Completion of online credit education correlates with more responsible credit card usage, however the experiments don t prove that the education causes this behavior. Meta-analysis: Summary: N/A Completed by US Bank, the study aimed at understanding intervention aimed at non-delinquent but high risk college students. Working Paper C 35