Foreign Language in the Elementary School Program

Foreign Language in the Elementary School Program Year One Interim Evaluation Report SY 008-09 Appendices September 009 Fairfax County Public Schools Office of Program Evaluation

APPENDIX A FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL IMPLEMENTATION BY SCHOOL AND GRADE

Foreign Language in the Elementary School Implementation by School and Grade School Language Grades Implementing in SY 008-09 Additional Grades Implementing in SY 009-10 Cluster 1 Chesterbrook Chinese 1, 3 Clearview Spanish 1, 3 Franklin Sherman Spanish 1,, 3 4 Kent Gardens French 1,, 3, 4, 5, 6 NA Cluster Cunningham Park Spanish 1, 3 Fairhill Chinese 1, 3 Graham Road Spanish 1, 3 Pine Spring Spanish 1, 3 Shrevewood Chinese 1,, 3, 4, 5 6 Wolftrap Chinese 1,, 3, 4 5 Cluster 3 Beech Tree Arabic 1, 3 Little Run Spanish 1, 3 Mantua Italian 3, 4, 5, 6 NA Sleepy Hollow Arabic 1, 3 Cluster 4 Belle View Spanish 1, 3 Gunston Japanese 1, 3 Mount Vernon Chinese 1 Woods Woodley Spanish 1, 3 Hills Cluster 5 Clermont Chinese 1,, 3 4 Cluster 6 Fairview Chinese 1 Sangster Spanish 1, 3 Terra Centre Spanish 1, 3 Cluster 7 Brookfield Spanish 1 Daniels Run Latin 3, 4, 5, 6 NA Greenbriar West Spanish 1, 3 Oak Hill Chinese 1, 3 Providence Chinese 1, 3 Latin 3, 4, 5, 6 (shifting 3 to Chinese) Willow Springs Spanish 1, 3 Cluster 8 Floris Spanish 1,, 3, 4 5 Navy Spanish 1, 3 Terraset Spanish 1,, 3, 4, 5, 6 NA Waples Mill Spanish 1 A-1

APPENDIX B FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL LITERATURE REVIEW

Foreign Language in the Elementary School Literature Review The purpose of this literature review is to provide a context for foreign language learning and describe effective program models and associated outcomes. The review provides the reader with an overview of the need for foreign language proficiency in the 1 st century, describes models for foreign language instruction, summarizes the qualities of successful programs, and reviews the literature regarding the impact of foreign language instruction on key outcomes: student academic performance and language proficiency. Need for Foreign Language Competence Periodically since the 1950 s with the launch of Sputnik and again during the height of the cold war in the early 1980 s, there have been calls for increased attention to foreign language instruction in the United States. Now in the 1 st century, globalization and the war on terror have again raised the call for creating a national framework to foster foreign language instruction and cross-cultural knowledge in the United States. In their paper, Building the Foreign Language Capacity We Need: Toward a Comprehensive National Language Framework, Jackson and Malone (009) argue that there are five pressing reasons to build the nation s language capacity: 1. National security and diplomacy The events of September 11, 001, highlighted the need for language proficiency and cultural knowledge to maintain U.S. security and to cultivate relations with a wider community of non-western countries. Certain languages were identified as critical languages and targeted for capacity building including Arabic, Persian, Indic and Turkic languages, Chinese, and Korean.. International commerce and economic development The National Committee for Economic Development issued a seminal report in 006 calling for increased foreign language skills and cultural awareness for students to maintain the United States as a global leader. The report identified several languages as critical to maintaining global economic competitiveness including Chinese, Japanese, Hindi-Urdu, Russian, and Portuguese. 3. Global perspectives and knowledge for all Americans In an increasingly interconnected world, businesses, even small businesses, are competing for clients internationally and domestically with different language and cultural backgrounds. Businesses have a competitive advantage if their staff can communicate in their clients native languages and understand their clients cultural backgrounds. 4. Needs of an increasingly diverse U.S. citizenry As diversity within the United States grows, service providers must be able to communicate in the languages spoken in the communities they serve. Medical professionals, law enforcement, fire and rescue workers, and social service providers, in particular, need language and cultural competence to perform their jobs. 5. Scholarship and research To maintain its leadership role in scholarship and research, the United States requires researchers who are able to access information about advances that are written in various other languages. Fairfax County, and the larger Washington D.C. metropolitan area, is a nexus for U.S. intelligence agencies, international organizations and businesses; hence, there is a great need to develop foreign language speakers who can address the needs of governments, businesses, and the non-profit sector. Additionally, it is a very diverse area with more than 5% of the residents of Fairfax County born in another country (13% Asian, 10% Hispanic or Latino, Metropolitan Washington Council of Governments, 006), and nearly one in five children in the Metropolitan Washington region speaking a B-1

Novice Intermediate Advanced Advanced Pre- Advance Intermediate Novice language other than English at home (Brookings Institution, 009). To deliver public services, the region needs to develop a cadre of service providers who can communicate with its linguistically and culturally diverse population. Models for Elementary Foreign Language Instruction There are many different models for teaching a foreign language in the United States. Programs begin at various grade levels and have different sequences and intensities of instruction, all leading to different outcomes. To aid policymakers in selecting a model to meet their proficiency goals, the American Council on the Teaching of Foreign Languages (ACTFL) published the ACTFL Performance Guidelines for K-1 Learners. The guidelines suggest the level of language performance that can be expected for various sequences of instruction beginning at different grade levels (Figure B-1). Descriptors are based on information gathered from foreign language professionals representing a variety of program models and articulation sequences. Descriptors are appropriate for languages more commonly taught in the United States. Descriptors assume a sustained sequence of standards-based, performance-outcomes, language instruction. The figure suggests that for Fairfax County Public Schools (FCPS) to meet its goal that students graduate with communicative competence in a language other than English, instruction needs to begin early in elementary school and be articulated through middle and preferably high school. Students who achieve at the intermediate-mid or higher are considered communicatively competent in the language. Figure B-1 ACTFL Performance Guidelines for K-1 Learners K-4 K-8 K-1 5-8 5-1 7-1 9-10 9-1 The remainder of this review focuses on foreign language instruction for elementary-aged students. There are three primary models of foreign language instruction for elementary-aged students, Immersion, FLES, and FLEX (Center for Applied Linguistics, n.d., 006a, 006b, 008). The models and their variations can be distinguished by program goals, contact time, and frequency. B-

Immersion programs are designed for students to acquire general academic proficiency in home language and the second language. A secondary or corollary goal is language proficiency in the second language. Immersion programs involve the most contact time of all models. In general, a foreign language is the language used to teach content, not the subject of instruction. Immersion programs vary in intensity. In total immersion, students receive instruction in all content areas in the foreign language. In Partial Immersion, up to 50% of the subjects are provided instruction in the foreign language. In Two-Way Immersion, equal emphasis is given to English and a non-english language. In Two-Way Immersion, native speakers of the non-english language comprise one to two thirds of the class, with the remainder being native speakers of English. In all immersion programs, the proficiency goal is expressed as Intermediate to Advanced proficiency in a language other than English. FLES programs focus on language proficiency in the target language, an introduction to literacy in the home and target languages, building cultural knowledge, and connecting language learning to the content of the early grades curriculum. The scope and sequence of the FLES program model are taught by a qualified foreign language teacher. Students learn grammar indirectly. A bilingual/multilingual gradelevel teacher teaches the foreign language. The curriculum is locally-specific. Content-enriched FLES programs offer instruction on core content areas to reinforce the curriculum and to provide a meaningful context for learning the target language. Contact time varies by number of sessions each week and the number of minutes per session. The most effective FLES programs meet three to five sessions per week for 30 minutes each session (Genesee, Holobow, Lambert, & Chartrand, 1989; and Genesee, 000). The proficiency goal is expressed as advanced to fluent (in a language other than English) or functional fluency in a language other than English by the completion of the elementary program. FLEX programs serve as an introduction to language learning and instill an awareness of different cultures, an appreciation of language/culture study, and motivation for further language study. The FLEX model introduces learners to one or more languages. After completing a FLEX program, students go on to a beginning language program. Exposure to a single language may take place from one to several days a week over six to nine weeks. The proficiency goal is expressed as novice to basic. Best Practices in Elementary Foreign Language Instruction A summary of the most cited studies of domestic and international foreign language programs suggests a relatively short list of best practices for FLES and other foreign language models (Christian, Pufahl, and Rhodes, 004, 005; DiFino & Lamardino, 004; Genesee, Holobow, Lambert, & Chartrand, 1989; Genesee, 000; Gilzow and Rhodes, 000; Kretschmer & Kretschmer, 1998; Met, 004; Pufahl, Rhodes, & Christian, 000; Thanasoulas, 001; and Torres, 1998): A well-articulated, coherent framework for instruction and assessment including curriculum designed to meet ACTFL s five standards for foreign language instruction (i.e., five C s communication, cultures, connections, comparisons, and community); Defined benchmarks for student performance and standards for teaching with regular student assessment, and program evaluation; Alignment between type of program selected and goals for language learning and strong schoolbased leadership; Authentic tasks, enjoyable and meaningful lessons, taught through core instructional content (e.g., mathematics, social studies, science), with sufficient intensity to motivate students; An early start, beginning instruction in early elementary and articulating instruction through middle and high school; B-3

Instruction using multiple modalities with an understanding of differences in learning styles and capacities; Sufficient contact time to ensure foreign language learning four or five days per week for 60-90 minutes per session; Defined target student audience while providing accessibility for all students; and Well-trained foreign language teachers and well-qualified teachers who receive rigorous teacher education and regular professional development. A best practice is a guide or benchmark. While there is some consistency among researchers about best practices for foreign language instruction, there is insufficient research data to conclude, without reservation, the extent to which a particular practice or feature (e.g., collaborative teaching, specific contact time and frequency, etc.) contributes to optimal learning in elementary foreign language instruction. Nonetheless, most of the practices outlined above appear reasonable, and the preponderance of the evidence across studies supports their importance for developing programs that will lead to proficiency in another language. Outcomes Associated with Effective Elementary Foreign Language Instruction In general, foreign language instruction has been associated with the following positive categories of outcomes: general academic proficiency through integrative instruction, second language acquisition, personal and cultural growth, and improved cognitive development. Armstrong and Rogers (1997), Cooper (1987), Curtain and Pesola (1994), Saunders (1998), and Taylor (004) concluded that elementary students with foreign language instruction outperformed their peers in other subjects and on standardized English and mathematics tests. In studies conducted by Boyson (1997), Boyson and Thompson (1998), and Harris, Hart & Met (1998), elementary foreign language students and English as a Second Language (ESL) students attained foreign language proficiency and/or demonstrated increased knowledge of concepts in other subjects. Donato and Terry (1995), Dumas (1999), Garfinkel and Tabor (1991), and Genesee (1985) observed personal and cultural growth for students in elementary foreign language programs. Dumas (1999) and Mechelli, et al. (004) show that learning a second language increases the density of grey matter that most likely relates to brain organization, an effect with multiple benefits. In addition, FLES programs have generated specific outcomes that can be attributed to their specific designs and implementations. FLES programs that include content integration have been associated with academic achievement in multiple subjects, higher standardized test scores, and benefits for regular and specific student subpopulations. For instance, in a study that many in foreign language instruction consider a seminal work, Curtain & Pesola (1994) reported that FLES students attained higher academic achievement in other subjects such as reading, social studies, and mathematics, performed better on SATs and ACTs, especially verbal, and experienced no loss in basic skills. These outcomes appear to be consistent across different quality implementations and modalities. Additionally, in a study of more than 144 FLES models, Robinson (199) concluded that FLES students outperform students without a foreign language on standardized tests and tests of basic skills in English, mathematics and social studies. In Cooper s study (1987), FLES students achieved higher SAT verbal scores than the control group, and FLES students, with multiple years of foreign language instruction, performed better on SAT subtests than students who had less foreign language instruction. In one of the few comprehensive, longitudinal studies on the impacts of FLES, Taylor, Feyten, Meros, and Nutta (008), Taylor (004), and Taylor-Ward (003) found that FLES teachers, who reinforce other content skills despite model variations, contributed to FLES student success at each grade level in outperforming control peers on English language arts, mathematics, science, and social studies tests. B-4

Third grade FLES students achieved statistically significant gains in mathematics and home language even FLES students with less mathematics instruction outperformed non-fles students in mathematics (Armstrong & Rogers, 1997). The FLES students received mathematics content in a foreign language, three sessions per week for 30 minutes each session, with varying intensities of all modalities. Saunders (1998) findings were similar. FLES students who received foreign language instruction five days a week for 30 minutes each session for four years significantly outperformed a control group that had no foreign language and was one year older on standardized mathematics assessments. Some FLES research has documented the benefits of the program for certain subpopulations of students. Harris, Hart, and Met (1998) found that daily FLES classes, with FLES in the core curriculum, reinforced concepts in other subjects for ESL and other at-risk students. In FLES and Immersion programs, Dumas (1999) concluded that race, gender, and non-language academic level did not intervene on the achievement of foreign language students in the English section of the state s basic skills test. The majority of the literature on FLES programs has focused on the benefits of foreign language learning for student academic achievement. Less research has focused on the language outcomes of FLES students. This may be due in part to the different levels of instructional intensity offered by various programs, so there is not a consistent benchmark for expected performance. Additionally, there is a lack of valid and reliable tests for school districts to measure progress. Assessment is also time intensive since it must include conversational and writing proficiency. The few studies in the literature that have focused on gaining proficiency in the target language have found that students do gain expected levels of proficiency in the target language (Boyson, 1997; Boyson & Thompson, 1998) vocabulary knowledge and conversational knowledge (Bernards Township Public Schools, 004). Conclusions In an increasingly global and flat world (Friedman, 005), the need for an educated citizenry with the ability to speak a language other than English and understand other cultures has risen to the forefront of national concern. A renewed focus has been placed on implementing foreign language programs in the public schools, beginning as early as kindergarten (Branaman & Rhodes, 1998). To develop the levels of proficiency required, ACTFL recommends a course of language study beginning in elementary school. Almost half of the foreign language programs in elementary schools follow a FLES model which focuses on developing basic conversational proficiency, introductory literacy, cultural knowledge, and in the case of content-embedded programs, instruction in core content areas (Branaman & Rhodes, 1998). Although FLES programs vary widely in the amount of instructional time they provide, research suggests that, in general, FLES programs improve achievement in core academics and develop students language proficiency in the target language. Based on the literature and practice knowledge, a core set of best practices has been developed; however, careful research has not detailed how these core practices are related to achieving different levels and types of outcomes (e.g., academic achievement, language proficiency, and cultural knowledge). In general, it is expected that the strongest programs will have a well-articulated curriculum tied to core content with regular assessment, engaging instruction of at least 75 to 10 minutes a week, and ongoing professional development for teachers. B-5

References American Council on the Teaching of Foreign Languages, (1998). ACTFL Performance Guidelines for K- 1 Learners. American Council on the Teaching of Foreign Languages, Alexandria, VA. Armstrong, P. W., & Rogers, J. D. (1997). Basic skills revisited: The effects of foreign language instruction on reading, math and language arts. Learning Languages, (3), 0-31. Bernards Township Public Schools. (004). K-5 world language program evaluation. Basking Ridge, NJ: Author. Retrieved August 5, 009, from http://www.bernardsboe.com/bernardsboe/cmfiles/docs/curriculum/k- 5%0World%0Lang%0Prog%0Eval%0005.pdf Boyson, B. (1997). Listening and speaking assessment of third grade students in the Georgia elementary school foreign languages model program. Washington, DC: Center for Applied Linguistics. Boyson, B., & Thompson, L. (1998). Student oral proficiency assessment of fifth grade students in the Georgia elementary school foreign languages model program. Washington, DC: Center for Applied Linguistics. Branaman & Rhodes. (1998). Foreign Language Instruction in the United States: A national survey of elementary and secondary schools. Center for Applied Linguistics; Washington, DC. Brookings Institution, The. (009, January). Demographic and economic trends in the national capital region and their effects on children, youth and families. Washington, DC: Metropolitan Washington Council of Governments. Center for Applied Linguistics. (n.d.). Definitions of immersion programs. Washington, DC: Author. Retrieved August 6, 009, from http://www.cal.org/resources/immersion/ Center for Applied Linguistics. (006a). Directory of foreign language immersion programs in U.S. schools. Washington, DC: Author. Retrieved August 6, 009, from http://www.cal.org/resources/immersion/ Center for Applied Linguistics. (006b). Directory of two-way bilingual immersion programs in the U.S. Washington, DC: Author. Retrieved August 6, 009, from http://www.cal.org/twi/directory/ Center for Applied Linguistics. (008). Bibliography of two-way immersion research. Washington, DC: Author. Retrieved August 6, 009, from http://www.cal.org/twi/bib.htm Center for Advanced Research on Language Acquisition. (n.d.). Bibliography of foreign language immersion research. Minneapolis, MN: Author. Retrieved August 6, 009, from http://www.carla.umn.edu/immersion/bibliographies/one-way.html Christian, D., Pufahl, I. U., & Rhodes, N. C. (004). Language learning: A worldwide perspective. Educational Leadership, 6(4), 4-30. Christian, D., Pufahl, I., & Rhodes, N. C. (005). Fostering foreign language proficiency: What the U.S. can learn from other countries. The Phi Delta Kappan, 87(3), 6-8. Retrieved August 11, 009, from JSTOR database. Committee for Economic Development (006). Education for Global Leadership: The Importance of International Studies and Foreign Language Education for U.S. Economic and National Security B-6

Committee for Economic Development, Washington, D.C. Cooper, T.C. (1987). Foreign language study and SAT-verbal scores. The Modern Language Journal, 71(4), 381-387. Retrieved August 11, 009, from JSTOR database. Curtain, H., & Pesola, C. A. B. (1994). Language and children: Making the match. White Plains, NY: Longman. DiFino, S., & Lamardino, L. (004). Language and learning disabilities: The ultimate foreign language challenge. Foreign Language Annals, 37(3), 399. Donato, R., & Terry, R. M. (1995). Foreign language learning: The journey of a lifetime. Lincolnwood, IL: National Textbook Company. Dumas, L. S. (1999). Learning a second language: Exposing your child to a new world of words boosts her brainpower, vocabulary, and self-esteem. Child, (February), 7, 74, 76-77. Friedman, T. (005). The world is flat: A brief history of the twenty-first century, Farrar, Straus, Giroux, New York: New York. Garfinkel, A., & Tabor, K. E. (1991). Elementary school foreign languages and English reading achievement: A new view of the relationship. Foreign Language Annals, 4(5), 375-38. Second Language Learning through Immersion: A Review of U.S. Programs Author(s): Fred Genesee. Source: Review of Educational Research, Vol. 55, No. 4 (Winter, 1985), pp. 541-561. Genesee, F., Holobow, N. E., Lambert, W. E., & Chartrand, L. (1989). Three elementary school alternatives for learning through a second language. The Modern Language Journal, 73(3), 50-63. Retrieved August 11, 009, from JSTOR database. Genesee, F. (000). Brain research: Implications for second language learning. Washington, DC: ERIC Clearinghouse on Languages and Linguistics [Center for Applied Linguistics]. Gilzow, D. F., & Rhodes, N. (000). Establishing high-quality foreign language programs in elementary schools: Perspectives on policy and practice. Providence, RI: Northeast and Islands Regional Educational Laboratory. (ERIC Document Reproduction Service No. ED46188) Harris, J., Hart, A., & Met, M. (1998). What are the advantages and disadvantages of FLES, FLES, and Immersion: An assistant superintendent and curriculum specialist view the models. In M. Met (Ed.), Critical issues in early language learning: Building for our children s future (pp. 11-1). Glenview, IL: Scott Foresman-Addison-Wesley. Jackson, F. H., & Malone, M. E. (009, April). Building the foreign language capacity we need: Toward a comprehensive strategy for a national language framework. Washington, DC: Center for Applied Linguistics. Retrieved August 9, 009, from http://www.cal.org/resources/languageframework.pdf Kretschmer, R., Jr., & Kretschmer, L. (1998). Is foreign language education for all learners? In M. Met (Ed.), Critical issues in early language learning: Building for our children s future (pp. 65-68). Glenview, IL: Scott Foresman-Addison-Wesley. Mechelli, A., Crinion, J. T., Noppeney, U., O Doherty, J., Ashburner, J., Frackowiak, R.S., et al. (004). Neurolinguistics: Structural plasticity in the bilingual brain. Nature 431(7010), 757. B-7

Met, M. (004). Improving students' capacity in foreign languages. The Phi Delta Kappan, 86(3), 14-18. Retrieved August 11, 009, from JSTOR database. Metropolitan Washington Council of Governments. (006). Fairfax County and the Washington region: A look at economic and demographic characteristics. Washington, DC: Author. Pufahl, I., Rhodes, N. C., & Christian, D. (000). Foreign language teaching: What the United States can learn from other countries. Washington, DC: ERIC Clearinghouse on Languages and Linguistics [Center for Applied Linguistics]. Robinson, D. W. (199). The Cognitive, Academic and Attitudinal Benefits of Early Language Learning. In Met, M., ed. Critical Issues in Early Language Learning. White Plains, NY: Longman. Saunders, C. M. (1998). The effect of the study of a foreign language in the elementary school on scores on the Iowa Test of Basic Skills and an analysis of student-participant attitudes and abilities. Unpublished doctoral dissertation, University of Georgia, Athens. Taylor, C. (004). The relationship between elementary school foreign language study in grades three through five and Louisiana students' academic achievement on standardized tests. Learning Languages, 10(1), 16-18. Retrieved August 10, 009, from Education Research Complete database. Taylor, G., Feyten, C., Meros, J., & Nutta, J. (008). Effects of FLES on reading comprehension and vocabulary achievement: A multi-method longitudinal study. Learning Languages, 13(), 30-37. Retrieved August 5, 009, from Education Research Complete database. Taylor-Ward, C. (003). The relationship between elementary school foreign language study in grades three through five and academic achievement on the Iowa tests of basic skills (ITBS) and the fourth-grade Louisiana educational assessment program for the 1st century (LEAP 1) test. Unpublished doctoral dissertation, Louisiana State University, Baton Rouge. Retrieved August 13, 009, from http://etd.lsu.edu/docs/available/etd-1103103-111054/unrestricted/taylor-ward_dis.pdf Thanasoulas, D. (001). The importance of teaching culture in the foreign language classroom. Radical Pedagogy, 3. Retrieved August 6, 009, from http://radicalpedagogy.icaap.org/content/issue3_3/7- thanasoulas.html Torres, R. (1998). Is foreign language education for all learners? In M. Met (Ed.), Critical issues in early language learning: Building for our children s future (pp. 60-61). Glenview, IL: Scott Foresman- Addison-Wesley. B-8

APPENDIX C FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL SITE-LEVEL PROGRAM MODEL

Foreign Language in the Elementary School Site-Level Program Model Resources Activities Outcomes* FLES Teachers Classroom Teachers FLES instructional materials FLES teacher laptops FLES lesson delivery Contributing to the FLES Blackboard community Collaborative planning between FLES and classroom teachers Professional Development Short-Term Student outcomes: Increased selfesteem Fearlessness of language Cultural awareness Flexible thinking Other outcomes: Teacher cultural awareness Intermediate Student outcomes: Near-native pronunciation Academic achievement Other outcomes: Lowered need for planning Consistent implementation Long-Term Student outcomes: Foreign language achievement at Novice level *Bolded outcomes represent critical program outcomes investigated in this program evaluation. Other outcomes: Consistency of student outcomes C-1

APPENDIX D FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL EVALUATION DESIGN

PROGRAM EVALUATION SUMMARY I. GENERAL PROJECT INFORMATION Project/Program Title: Project/Program Contact and Department: Comprehensive Evaluation Projected Start Date: Comprehensive Evaluation Projected Completion Date: Foreign Language in the Elementary School (FLES) Paula Patrick, ISD August 008 October 011 Expected Deliverables: Interim Report (Year 1) October 009 Interim Report (Year ) October 010 Final Report (Year 3) October 011 OPE Lead Evaluator(s): Evaluation Team Members: Source of Request for the Evaluation: Background/History: Purpose of the Evaluation: Lidi Hruda Marty Smith, Cluster I Tonya Cox, Gunston ES Steve Gossin, Cherry Run ES Paula Patrick, ISD Robert Harrison, Glasgow MS Marie Hinton, ISD Recardo Sockwell, DA Chantal Follett, DA School Board/Superintendent The purpose of the FLES program is to provide elementary students with learning opportunities in a language other than English. It is one of several FCPS offerings at the elementary level that support the School Board s Student Achievement Goal 1. ( communicative competence in one or more languages other than English by high school graduation ). The School Board goal has generated increased interest in potentially expanding the program to all FCPS elementary schools to make FLES a universal aspect of elementary instruction within FCPS. To provide the FCPS School Board and Leadership Team with evidence-based judgments about the effectiveness of the program s model and its implementation in developing communicative competence in students, particularly in comparison to other possible models of elementary instruction (i.e., immersion). D-1

II. PROPOSED PROGRAM EVALUATION DESIGN Project/Program Title: OPE Lead Evaluator(s): Foreign Language in the Elementary School (FLES) Lidi Hruda Evaluation Questions 1a. To what extent is the FLES programmatic approach designed to contribute at the elementary level to Student Achievement Goal 1.? 1b. To what extent is the FLES program designed to meet the needs of external forces (i.e., business community, government, etc.)? Data Collection Data Collection Data Source Method Time Line REQUIRED PROGRAM COMPONENTS I. Program Definition Program manager Interview Year 1 (Spring 009) External studies and Literature Review (Years and 3, only evaluations of FLES if program definition programs changes) Program Documents Record Review Websites and documents of external stakeholders Record Review Data Analysis Meta-analytic work on FLES best practice and qualitative comparison to FCPS FLES model. Year 1 (Spring 009) Quantitative rating based on standard Program Definition criteria used in FCPS program evaluations Reporting Time Line Interim Report, Year 1 (Fall 009) Final Report, Year 3 (Fall 011). To what extent do participating sites implement the program as defined by central office? FLES Classes Program Documents about Fidelity of Implementation or used for Monitoring System FLES Teachers Regular Classroom Teachers Principals FLES Blackboard Site Program manager II. Fidelity of Implementation Classroom Year 1 (Spring 009) Observations Year (Spring 010) Record Review Year 3 (Spring 011) Survey or focus group Survey or focus group Survey Record Review Interview Quantitative and qualitative comparisons of data to expectations of program model. Quantitative rating based on standard Fidelity of Implementation criteria used in FCPS program evaluations Interim Report, Year 1 (Fall 009) Interim Report, Year (Fall 010) Final Report, Year 3 (Fall 011) D-

Evaluation Questions 3a. To what extent do students participating in FLES meet the selected standards described by the American Council on the Teaching of Foreign Language (i.e., Communication, Cultures, Connections)? 3b. Do some groups of students demonstrate stronger achievement benefits from FLES than others (i.e., LEP, Special Education)? Data Source Data Collection Method Data Collection Time Line III. Outcomes Grade 3 and 5 Foreign Standardized testing Year (Spring 010) Language PALS Year 3 (Spring 011) Grade 4 and 5 SOL Standardized testing Year 3 (Spring 011) Cultural Awareness Measure Survey Year 3 (Spring 011) Principals Survey Year 1 (Spring 009) (Years and 3, only if needed) FLES Teachers Survey Year 1 (Spring 009) Year (Spring 010) Year 3 (Spring 011) Regular Classroom Teachers Survey Year 1 (Spring 009) Year (Spring 010) Year 3 (Spring 011) Data Analysis Descriptives of FLES student performance Year (Spring 010) Year 3 (Spring 011) Quantitative comparisons between FLES students and Immersion students (FL, SOL, and culture) Year (Spring 010) Year 3 (Spring 011) Quantitative comparisons between FLES students and elementary students not receiving FL instruction (SOL and culture) Year 3 (Spring 011) Reporting Time Line Interim Report, Year (Fall 010) Final Report, Year 3 (Fall 011) Quantitative rating based on standard Outcomes criteria used in FCPS program evaluations Year (Spring 010) Year 3 (Spring 011) D-3

Evaluation Questions 4. To what extent does the FLES program maximize cost-effectiveness, particularly in comparison to elementary Immersion programs? Data Collection Data Collection Data Source Method Time Line IV. Cost Analysis Budget Documents Record Review Year 1 (Spring 009) ISD Budget Liaison Interview Year (Spring 010) Year 3 (Spring 011) Program Manager Interview Program Documents about Funding Principals Record Review Survey Data Analysis Quantitative and qualitative descriptions of funding (including external sources) and cost optimization level Year 1 (Spring 009) Year (Spring 010) Year 3 (Spring 011) Cost-effectiveness comparison of FLES and Immersion Year (Spring 010) Year 3 (Spring 011) Reporting Time Line Interim Report, Year 1 (Fall 009) Information provided on cost description and optimization; no rating of cost-effectiveness in Year 1 Interim Report, Year (Fall 010) Final Report, Year 3 (Fall 011) 5. To what extent does the FLES program have procedures in place to monitor program implementation and student progress? Outcome data NA already collected for other questions SUPPLEMENTAL PROGRAM COMPONENTS A. Monitoring System Program manager Interview Year 1 (Spring 009) Principals Survey Year (Spring 010) FLES Teachers Survey or focus Year 3 (Spring 011) group Program Documents Record review used for Monitoring System Quantitative rating based on standard Program Costs criteria used in FCPS program evaluations Year (Spring 010) Year 3 (Spring 011) Qualitative analysis of data; Quantitative rating based on standard program quality rating scale used in FCPS program evaluations Year 1 (Spring 009) Interim Report, Year 1 (Fall 009) Interim Report, Year (Fall 010) Final Report, Year 3 (Fall 011) D-4

Evaluation Questions 6. To what extent do professional learning opportunities offered to FLES and classroom teachers support FLES program implementation at schools? Data Source Data Collection Data Collection Method Time Line B. Professional Development Program manager Interview Year 1 (Spring 009) Principals Survey Year (Spring 010) FLES Teachers Survey or focus Year 3 (Spring 011) group Regular Classroom Survey or focus Teachers group Program Documents Record review about professional learning and training Data Analysis Qualitative analysis of data; Quantitative rating based on standard program quality rating scale used in FCPS program evaluations Year 1 (Spring 009) Year (Spring 010) Year 3 (Spring 011) Reporting Time Line Interim Report, Year 1 (Fall 009) Interim Report, Year (Fall 010) Final Report, Year 3 (Fall 011) 7a. To what extent does the process used to identify FLES sites and associated languages optimize articulation among school levels, student outcomes, and program costs? 7b. To what extent are site/language identification criteria used uniformly across schools? C. Site Identification Program manager Interview Year 1 (Spring 009) Custer offices Survey (Years and 3, only Program Documents Record Review if staffing changes) about site selection Cluster offices Survey or interview Program Documents Record review about site selection Qualitative analysis of data; Quantitative rating based on standard program quality rating scale used in FCPS program evaluations Year 1 (Spring 009) Interim Report, Year 1 (Fall 009) Final Report, Year 3 (Fall 011) D-5

APPENDIX E FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL ADDITIONAL DETAILS ABOUT PROGRAM DEFINITION

Additional Evaluation Evidence Foreign Language in the Elementary School Additional Details about Program Definition When the Department of Instructional Services (ISD) recommended to the School Board that the county adopt the Foreign Language in the Elementary School (FLES) model to support their Student Achievement Goal (SAG) 1. ( all Fairfax County Public Schools graduates communicate in at least two languages ), the recommendation was based on the belief that the FLES model was the most economical and feasible way to start students on the path to communicative competence in a foreign language. The FLES model exposes all students in participating elementary schools to at least one hour of foreign language instruction per week, starting in grade 1 and continuing through the end of elementary school (typically grade 6, but grade 5 at some schools). In contrast, while serving students in the same grades as FLES, the Partial Immersion (PI) program is an optional program where select participating students learn science and math content in the targeted foreign language. Attrition from the PI program tends to be high by the upper grades. Therefore, while the PI program offers more intensive language instruction, it reaches fewer students than FLES; and participating students do not necessarily continue to receive foreign language instruction through the end of elementary school. [Fairfax County Public Schools (FCPS) also has one additional program, Two- Way Immersion, offered only at the kindergarten level and, thus, not targeting the same population of students as FLES.] Beginning foreign language instruction at the elementary level school is intended to set students on a path to develop communicative competence in another language by the end of high school, thus achieving SAG 1.. More specifically, according to FLES assessment rubrics, by the end of grade 6, participating students are expected to have near native pronunciation and be able to reach the American Council for Teaching Foreign Language (ACTFL) Performance Guidelines for K-1 Learners. These are defined within FCPS as being able to write and speak the target language at a high-novice level (3 on a 9-point rubric, with 9 representing highly advanced achievement in the target language). 1 These expectations are supported by external research findings that students in high quality FLES programs can achieve language proficiency after three years of instruction leading to fluency in listening and speaking (Boyson, 1997). According to the program model described by FLES staff during the exploratory evaluation, FCPS students receiving FLES instruction are also expected to demonstrate increased interest in learning languages that motivates them to engage in continued foreign language study at the secondary level. The FLES program model is set up around achieving the expected levels of foreign language competence by providing students with at least twice-weekly lessons in the target language. Many of the elements identified by FLES program staff as necessary to a strong FLES lesson are intended to motivate students and encourage their engagement in the lesson. These instructional practices include using multiple modalities (aural, body-kinesthetic, oral, visual, written), active observation of the FLES lesson by the regular classroom teacher to model engagement in foreign language learning, and embedding the foreign language instruction within a content-based framework that provides students foreign language vocabulary that is immediately of use. A stated goal of FLES is to prepare FCPS students to compete in the 1 st century. Specifically, FLES is expected to improve cross-cultural understanding and prepare students for jobs in the coming decades. In 006, a seminal report on workforce readiness was released that surveyed 431 human resource officials 1 Adjustments in writing expectations are made for students learning non-roman script language (i.e., Arabic, Chinese, and Japanese). Students studying these languages are expected to achieve Novice-mid levels in writing, rather than Novice-high as is true for students studying Roman script languages. No adjustment is made to the expectations for speaking, so all languages have identical expectations. E-1

about the skills and knowledge needed for the 1 st century. Sixty-three percent of respondents rated the ability to speak a foreign language as the fastest growing need in the workplace over the next five years. In follow-up interviews, respondents noted that the ability to speak a foreign language is critical for understanding other cultures, working cooperatively in global teams, and competing in today s global markets. Other organizations have also called for the need to develop a workforce that speaks foreign languages. The Workforce Investment Council (008) lists language instruction as one of ten strategic opportunities because of the perceived growth in need of a workforce with proficiency in multiple languages. Thus, the FLES program s intention to encourage continued study of foreign language in secondary school by students and foreign language competence by high school graduation is aligned with the business community needs reported in surveys and interviews. In 003, the federal government launched the National Security Language Initiative (NSLI) to identify and create training opportunities for high-need or strategic languages. The Secretaries of State, Education, and Defense, and the Director of National Intelligence (DNI), partnered together and identified Arabic, Chinese, Japanese, Korean, Russian languages and the Indic, Persian, and Turkish language families as critical languages to support and to achieve national security and foreign policy goals. The Department of Education developed a grant program called the Foreign Language Assistance Program (FLAP) to fund instruction in these languages. Currently, three of FCPS FLES schools (1 Arabic, Chinese) are supported by funds from these federal grants; so obviously, some FCPS schools have aligned their language selection with publicly acknowledged government needs. However, looking more closely at NSLI recommendations, a minority of FCPS FLES schools have a target language identified by federal officials as critical ( Arabic, 9 Chinese, 1 Japanese), while nearly two-thirds have a target language not considered critical by government officials (1 French, 1 Italian, 1 Latin, 17 Spanish). Currently, the critical languages not found at the elementary level are also not available at the secondary level in FCPS, undoubtedly limiting selection of any of these additional critical languages as the target language at a FLES school. Nonetheless, FCPS may wish to consider more directly targeting implementation of the identified critical languages (even if only limited to Arabic, Chinese, and Japanese) should FLES expand into additional schools in the future. In a 000 policy brief, Douglas Gilzow and Nancy Rhodes summarized nine characteristics of highquality elementary foreign language programs. Comparing the FLES program to these nine criteria indicates a mixture of strengths and weaknesses. In line with three of the suggested criteria, FLES lesson delivery aims to give all students enjoyable, meaningful language instruction tied to content and based on a curriculum focused around the five national foreign language standards endorsed by ACTFL. FLES site-level activities also include collaboration between FLES and classroom teachers, matching another of the characteristics. At the program level, an additional criterion is matched as program staffs recruit and provide regular professional development to qualified teachers. While not specified in the program model, the program s application process also includes checks to ensure articulation between elementary school through to middle and high school, meeting a sixth criterion of quality elementary foreign language instruction. However, three of the quality characteristics prove more problematic for the FLES model. The first is a mismatch between the suggested intensity of instruction and the amount provided by FLES. While the FLES program model targets at least two lessons per week lasting 30 minutes each, no participating schools provide more FLES instruction than the minimal level. However, the criteria suggested, for a quality program, target 3 to 5 lessons per week for no less than 30-40 minutes. Thus, on a weekly basis, FLES students receive 1 hour of instruction, as compared to the recommended 1.5 to 3.3 hours. This results in an instructional gap of approximately 18 to 85 hours over the course of a single year or 100 to 500 hours over the course of six elementary grades. Thus, the FLES program provides much less exposure to the target language than ACTFL recommends. However, while making sense to have more The elementary school that is phasing out Latin and phasing in Chinese is counted here as implementing Chinese as the target language. E-

frequent instruction to achieve good outcomes, it should be remembered that the recommended amount of instruction is not validated by research, so a lower dosage of instruction may be as successful. The FLES program also does not meet the two interrelated criteria of clearly stated goals at each grade level and regular evaluation of student learning. The program supports two summative division-wide assessments, at the end of grades 3 and 6, with specific detailed benchmarks aligned with expectations for language proficiency set forth by ACTFL and documented via assessment rubrics. However, there are no common assessments systematically administered before these grades to assess student progress toward meeting expectations, nor is there any type of formal or systematic assessment used other than informal assessments teachers engage in during the course of lessons. Component Analysis, Judgment, and Rating Prerequisites Broad rationale for program: Overall purpose described in relation to division and/or national context Broad description of target population and need for the program Broad description of program activities: Site level Central level Broad description of meaningful and measurable outcomes Broad description of resources used to support program activities Standard Elements 3 1. The rationale for the program is documented and includes: a. A research basis b. Evidence of the program s unique contribution to FCPS in terms of activities and/or targeted population c. A description of need for the program and fit within FCPS culture. Program activities are documented and include: a. Clear and explicit descriptions of site-level activities b. Clear and explicit descriptions of central-level activities 3. Program resources are clearly documented 3 and purposefully aligned with activities. 4. Meaningful and measureable short, intermediate, and long-term outcomes are documented and include: a. Associated timelines b. A clear relationship to program activities c. Standards and benchmarks 5. There is a common understanding of program rationale, activities, resources, and outcomes among stakeholders. As with all program quality components, the first step in gauging program functioning so that OPE can assign a rating, is to determine whether the program has met all Prerequisites. For the Program Definition component, FLES documents and artifacts clearly indicate that the program has done this successfully. For example, documents and artifacts such as the FLES webpage on www.fcps.edu and PowerPoint presentations the program has posted on FCPS 4/7 document Program Definition information regarding the program s rationale, purpose, resources, activities, and outcomes. In fact, in documenting this information and providing the level of detail that is found in these program-generated artifacts, the FLES program reaches into the Standard Elements. 3 Documentation referred to in the Program Definition s Standard Elements means program-generated documents and artifacts that exist separate from documentation created by or collated by the Office of Program Evaluation (OPE) for the exploratory or comprehensive evaluation processes. E-3

Moving on to the Standard Elements, Standard 1 is partially but not fully met. The FLES program is a content-embedded approach to foreign language instruction adopted by the division to contribute to the School Board goal that all students should be communicatively competent in two languages by graduation. Literature about best practices for elementary foreign language instruction identify nine criteria of effective FLES programs, six of which, as discussed more extensively above, have been embedded into the program model and documented through various program documents and artifacts. Literature suggests that FLES Program Definition would be improved by modifying activities (amount of instruction, evaluation of student learning) and Outcomes (articulation of program goals at every grade level). Furthermore, multiple documents tie the FLES program s purpose to SAG 1. and specifically advocate for using the FLES program to offer all students an early foundation for achieving this goal. Program-generated documentation provides detailed expectations that meet Standard for both site and central office activities. The documents used to describe the program clearly state the expectations for implementation at the schools including the duration, characteristics of instruction, and required conditions at the schools to support FLES. The documents also lay out the expectations for teacher s participation in ongoing professional development led by central office staff. Central office staff engages in activities to ensure consistency and quality of the program, including overseeing instruction in the classroom and curriculum development. While central office activities are not as frequently documented as site-level activities, there is sufficient documentation to ensure clarity of roles and responsibilities. With respect to Standard 3, the program articulates required resources and aligns them to program activities. Budget-related documentation and PowerPoint presentations reflect that in the first year of FLES implementation at a school, teachers are hired at a.5 position to provide instruction in first (and frequently second grades). Classroom instructional time is purposefully minimized so teachers have time to develop lessons for the grade levels they are teaching and share them with their FLES teacher colleagues. It is anticipated that teachers will not have sufficient time to plan lessons as they begin to support more grades and up to 450 students. The program also aligns its resources to support consistent and quality instruction through monthly professional development meetings, classroom observations and curriculum development. Other program files link expenditures to professional development that supports strong FLES instruction by teachers. For Standard 4, the program has articulated short, intermediate and long-term outcomes, but has not consistently developed standards and benchmarks for these outcomes. For example, the program has articulated short-term goals of FLES instruction in the first month of instruction including fearlessness of language, increased self-esteem and cultural awareness, but has not established standards or benchmarks for these outcomes. The program has been more successful in establishing standards and setting expectations for intermediate and long-term goals to develop functional proficiency in a world language. They have set benchmarks for student performance in the target language in grade 3 and grade 6 based on national standards from ACTFL and are piloting student performance assessments on these standards. The program would also benefit by setting benchmarks for performance in the grades that do not have assessments to ensure that students are making sufficient progress to meet the benchmark. Although improving student achievement in the core content is a stated goal of the program, it does set expectations or monitor student performance in this area. Lastly, on Standard 5, the FLES program has been successful in developing a surface understanding of the program activities and outcomes among principals and most classroom teachers; however, the rationale linking activities to outcomes was not as clearly understood. Principals and classroom teachers consistently articulated the goals of the program and some of the activities that support reaching the goals, but some schools supported conditions that violated the program model. For example, during SY 008-09, some schools did not schedule joint teacher planning time to support site-level collaboration between FLES and classroom teachers, or few schools provided a separate classroom for FLES, rather than having FLES teachers come to the students regular classrooms. The program would benefit from clearer E-4

articulation of not just the activities themselves, but harkening back to Standard 1, the rationale behind the activities and how adherence to the model is necessary to achieve intended student outcomes. Evaluation Questions and Responses To what extent is the FLES programmatic approach designed to contribute at the elementary level to SAG 1.? (Evaluation Question 1a) Summarizing the response to this evaluation question, the FLES program is the only program currently being implemented in the division that could contribute at the elementary level to all FCPS students meeting SAG 1.. The FLES program s overall purpose and instructional approach target this goal. The FLES program, as implemented in FCPS, is organized to meet six of the criteria: (1) engaging instruction; () curriculum based on the five ACTFL standards (Communities, Communication, Cultures, Connections to other subjects, and Comparisons with other languages and cultures); (3) inclusion of all students; (4) communication and coordination between FLES and classroom teachers; (5) native or near native speakers of the language who receive ongoing professional development; and (6) articulation through middle and high schools. The program needs to improve its defining of program goals for each grade level so that all grades are equally articulated, and to develop a regular process for assessing student learning in some way at all grade levels. While the preceding six characteristics of a quality elementary foreign language program are purposely reflected by the FLES model, the program currently does not meet three important characteristics. Should research validate the critical need for more instructional time, the small amount of FLES instructional time (35-40 hours per year) could prove a serious impediment to achieving the program s intended support for SAG 1.. The FLES program also demonstrates unevenness at some grade levels in defining clearly stated instructional goals and conducting regular assessments of student learning to monitor whether or not such goals are being achieved. To what extent is the FLES program designed to meet the needs of external forces (i.e., business community, government, etc.)? (Evaluation Question 1b) Overall, the FLES program s intention to encourage students to continue studying foreign language in secondary school and communicative competence in a second language by high school graduation is aligned with business and government needs. However, nearly two-thirds of FLES schools have a target language not considered critical by government officials (1 French, 1 Italian, 1 Latin, 17 Spanish 4 ), highlighting that the alignment of the program with the specific needs of external forces has room for improvement. 4 The elementary school that is phasing out Latin and phasing in Chinese is counted here as implementing Chinese as the target language. E-5

References American Council on the Teaching of Foreign Languages, (1998). ACTFL Performance Guidelines for K- 1 Learners. American Council on the Teaching of Foreign Languages, Alexandria, VA. Boyson, B. (1997). Listening and speaking assessment of third grade students in the Georgia elementary school foreign languages model program. Washington, DC: Center for Applied Linguistics. Gilzow, D. F., & Rhodes, N. (000). Establishing high-quality foreign language programs in elementary schools: Perspectives on policy and practice. Providence, RI: Northeast and Islands Regional Educational Laboratory. (ERIC Document Reproduction Service No. ED46188) Workforce Investment Council. (008). Washington, D.C. strategic plan for workforce development. Washington, DC: Author. Retrieved August 11, 009, from http://dcchamber.org/clientuploads/ WIC_StrategicPlan.pdf E-6

APPENDIX F FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL ADDITIONAL DETAILS ABOUT FIDELITY OF IMPLEMENTATION

Additional Evaluation Evidence Foreign Language in the Elementary School Additional Details about Fidelity of Implementation The Office of Program Evaluation (OPE) used multiple sources to gather evidence about site-level implementation of the program model. These sources included interviews with central office program staff; observations in Foreign Language in the Elementary School (FLES) classrooms; surveys of FLES teachers, classroom teachers, and principals; and record reviews of program documents and artifacts, including the FLES 4/7 Blackboard site. As described in the Year One Interim Evaluation Report SY 008-09, the focus of the evidence gathered was on the four specific site-level activities expected by the FLES program model, as described by program staff: 1. Delivering FLES lessons;. Contributing to the FLES Blackboard community; 3. Collaborating; and 4. Participating in program-specific professional development. All four activities are the responsibility of the FLES teacher(s); however, in some cases, the teachers must rely on the cooperation of others. For example, the FLES teacher is tasked with collaborating with classroom teachers, but this task is facilitated when classroom teachers are open to the collaborative process or when the school provides for joint planning time between the FLES and regular classroom teachers. Similarly, while lesson delivery is to be done within parameters such as two 30-minute lessons per week for each class, school administrators must find a way to fit the lessons into the schools master schedules. Delivering FLES Lessons FLES Lesson Characteristics Lesson delivery is the centerpiece of FLES implementation activities as it is the element that directly provides the foreign language instruction to students. FLES program staff provided a list of the primary characteristics of effective FLES lesson delivery, which included: Two lessons per week for 30 minutes each; Inclusion of all Special Education students in the regular classroom (Note: self-contained Special Education students may or may not receive FLES at school s discretion.); Multimodal delivery of lessons (oral, written, visual [pictures and manipulatives], songs, kinesthetic; expect to cover all five within a week); Instruction occurs in target language most or all of the time; FLES instruction occurs in the students regular classrooms; Both the FLES and regular classroom teacher remain in the classroom for FLES lessons; Regular classroom teacher actively observes (watches and listens to) FLES lessons; and Students respond orally or in writing to target language. There were some differences in the percent of grades 1 and classrooms implementation fidelity on expected lesson characteristics. For example, while all FLES teachers used at least three modalities at both grades, unsurprisingly, writing was a more common practice in grade (89%) than grade 1 (81%). F-1

This contrasted with the more frequent use of aural and kinesthetic instruction at grade 1 (aural: 87%; kinesthetic: 87%) than grade (aural: 75%; kinesthetic: 8%). Perhaps more importantly, the level of adherence demonstrated in grade 1 FLES lessons to having both the regular classroom teacher and FLES teacher in the room during the lesson (90%) and having the regular classroom teacher actively observe the lesson (55%), both showed declines by grade (75% and 46%, respectively), indicating declining trend of involvement in FLES, by the regular classroom teachers, as students get older. Similarly, the inclusion of all students in FLES lessons showed a marked decline between grade 1 (90%) and grade (75%), perhaps indicating declining commitment by schools to include all students as competing demands interfere in the later grades. Table F-1 Percent of FLES Lessons Adhering to Expected Characteristics (n=59 observations) Lesson Characteristic Grade 1 Grade Total 30-minute lesson Regular classroom and FLES teacher in room All students in room Regular classroom teacher actively observes Multiple modalities (oral plus at least additional modalities) - Oral - Written - Visual - Aural - Kinesthetic Instruction in target language Students respond to target language Instruction in regular classroom 6 84% 8 90% 8 90% 17 55% 31 31 5 81% 30 97% 7 87% 7 87% 9 94% 30 97% 9 94% 4 86% 1 75% 1 75% 13 46% 8 8 5 89% 7 96% 1 75% 3 8% 7 96% 8 6 93% 51 86% 49 83% 49 83% 30 51% 59 59 50 85% 57 97% 48 81% 50 85% 56 95% 58 98% 55 93% Table F-1 examines how implementation differed by FLES language. Across all languages, use of multiple modalities for instruction was high. FLES teachers of the Chinese language were less likely to use written language in instruction than teachers of other languages, and Arabic teachers relied less on aural language than other teachers. All teachers used oral language, and most used visuals and kinesthetic approaches in their teaching. As with the overall sample of all languages, the most frequently implemented lesson characteristic was multiple modalities. Also, mirroring the overall results, the most challenging lesson characteristic to implement for each target language was active observation by the classroom teachers. One Arabic classroom had a lower percent of teachers remaining in the room because one of the two schools teaching F-

Arabic used FLES time for teacher planning. Of the remaining Arabic classrooms, teachers actively observed instruction. FLES classrooms teaching Arabic also had the highest percent of English spoken. This may represent the amount of classroom management that was done in English at one school teaching Arabic that used 55-minute FLES instructional blocks once a week, rather than the recommended 30- minute blocks twice a week. In some cases, fidelity, with which a school implemented FLES, was directly related to school-based decisions, such as the previously mentioned master schedule that allots a longer lesson block and schedules FLES during teacher planning periods. The program collected FLES teacher schedules, which indicated that some FLES teachers had no transition time to move from classroom to classroom and still provide a 30-minute lesson. Further, contrary to recommendations found in literature for best practices with FLES, teachers frequently taught FLES lessons contiguously to different grade levels. In other cases, students were pulled from the classroom to receive other services rather than receive FLES instruction. Two schools also provided teachers with their own classrooms, rather than holding instruction in the regular classroom as stipulated by the FLES model. Table F- Implementation of FLES Instructional Characteristics by Language Lesson Characteristic Arabic Chinese French Japanese Spanish Total 30-minute lesson Regular classroom and FLES teacher in room All students in room Regular classroom teacher actively observes Multiple modalities (oral plus at least additional modalities) - Oral - Written - Visual - Aural - Kinesthetic Instruction in target language Students respond to target language Instruction in regular classroom Total 4 67% 4 67% 6 4 67% 6 6 6 6 33% 5 83% 5 83% 6 6 6 10% 1 9.3% 11 84.6% 13 6 46.% 13 13 7 54% 1 9% 11 85% 11 85% 13 13 11 85% 13 % 1 50% 1 50% 1 50% 0 0% 3.4% 1 50% 1 50% 3.4% 30 83.3% 31 86.1% 8 77.8% 18 50% 36 36 33 91.7% 35 97.% 31 86.1% 30 83.3% 34 94.4% 35 97.% 36 36 61.0% 50 84.7% 49 83.1% 49 83.1% 30 50.8% 59 59 50 84.7% 57 96.6% 48 81.4% 84.7 56 94.9% 98.3 55 93.% 59 Classrooms were categorized into three levels of implementation based on the number of FLES instructional elements that were present, low (five or fewer elements), moderate (six to seven elements) or full (eight elements). Of the 59 classrooms observed, the majority, 5.5% (n=31), were categorized as demonstrating a moderate level of implementation; 35.6% (n=1) were categorized as having full F-3

implementation and about 1% (n=7) were categorized as having low implementation (Table F-). At the school level, six schools were identified as implementing FLES instructional elements fully in all classrooms. Of the remaining schools, ten schools were rated as implementing the program moderately in all classrooms, and one was rated as having low implementation across the classrooms observed. The remainder of the schools, 11, demonstrated different levels of implementation across classrooms (Table F-3). Language Arabic Chinese French Japanese Spanish Total Table F-3 Implementation Levels Overall and by Language Implementation Level Low Moderate Full (5 or fewer (6 to 7 (8 elements) elements) elements) 1 1 4 16.7% 16.7% 66.7% 0 9 4 0% 69.% 30.8% 1 1 0 50.0% 50.0% 0% 0 0 0% 0% 5 18 13 13.9% 7 11.9% 50.0% 31 5.5% 36.1% 1 35.6% Total 6 13 36 59 FLES program staff expressed the belief that the impediment to full fidelity at sites is underlying beliefs about FLES instruction. For example, program staff reported that some schools choose not to adhere to the program s expectations that lessons be delivered in the student s regular classroom, choosing instead to have a separate FLES classroom that all students come to for FLES. Program staff believes that principals who are convinced by their FLES teachers to provide a separate classroom are operating under a belief system that does not match the FLES philosophy. When asked about challenges their schools experience implementing the FLES program, only one expectation was considered difficult by multiple administrators; approximately 5 percent of administrators reported that including all students in the FLES lesson was difficult for schools. An additional 10 percent of principals who currently felt their schools were doing a good job with lesson delivery expectations saw increasing challenges down the road as FLES expanded into more grades at the school owing to IEP requirements for receiving services such as OT or speech from itinerant teachers. FLES and classroom teachers rarely raised challenges that would indicate sites are not adhering to expectations for lesson delivery. Two classroom teachers indicated that they were unable to stay in the classroom during the FLES lessons, while one FLES teacher admitted difficulty maintaining the target language when handling student behavior. Interestingly, both FLES and classroom teachers frequently tied these two expectations together having the classroom teacher remain during the FLES lesson was typically viewed as easing the classroom management burden of the FLES teacher by allowing the classroom teacher to handle behavioral problems in English, individually with students, as needed, while the FLES teacher maintained speaking in the target language and delivering the FLES lesson. Related to implementation fidelity, multiple sources raised the issue of using the regular classroom, rather than a separate classroom, for FLES lesson delivery. Program staff acknowledged that a few schools had placed FLES teachers into their own separate classrooms, where students came for instruction. Survey F-4

responses from school-based administrators, FLES teachers, and classroom teachers indicated that while most schools adhere to having FLES delivery occur within the students regular classroom, some respondents did not understand why this expectation existed and preferred that it be eliminated from the program. Teachers lauded perceived benefits such as lower set-up time or greater access to target language materials to justify their suggestion. Nonetheless, the vast majority of schools are adhering to the expectation that FLES teachers come into students regular classroom to deliver FLES instruction. FLES Program of Studies Integration The FLES Program has its own Program of Studies (POS) that encompasses both foreign language and FCPS POS in the four core academic subjects (language arts, mathematics, science, and social studies). As can be seen in Table F-4, about 81 percent of lessons addressed a FLES POS. The most popular FLES theme for students in grade 1 was My Animal Habitats, while the most popular one for grade was Our Society (Table F-5). Table F-4 FLES Program of Studies FLES Theme Addressed Grade Level 1 Yes 5 3 5% 48% No 6 5 55% 46% Total 31 8 53% 48% Total 48 81% 11 19% 59 FLES Theme Grade 1/Grade Table F-5 Classroom Observation Grade Level 1 Total My School/School Community 4 16% My Family/Expanding 5 Communities 0% My Animals/Animal Habitats 10 40% My Daily Life/Our Society 6 35% Total 5 4 17% 3 13% 5 % 11 65% 3 8 17% 8 17% 15 31% 17 35% 48 Student Engagement Implied by standards for good instruction, as well as specifically cited by the American Council on the Teaching of Foreign Languages (ACTFL), is the need to engage students. Observation data gathered about levels of student engagement during FLES delivery were generally high, with an average of 87 percent of students engaged during the lessons. Almost 80 percent of students were highly or fully engaged during instruction (Table F-6). F-5

Very Low 0% 0 0% Table F-6 Observed Levels of Student Engagement Low 40% 3.4% Moderate 60% 10 16.9% High 80% 1 0.3% Full 35 59.3% Total 59 Additional analyses were conducted to investigate factors that might contribute to student engagement. A Pearson correlation was conducted to test whether there was a relation between the number of FLES lesson characteristics a program implemented and ratings of student engagement. The results indicated a marginal relation (r =.5, p=.06), indicating that there was a trend towards higher student engagement as fidelity with the expected lesson characteristics increased, but this trend was weak (Table F-7). A second analysis that focused solely on the number of modalities found a significant moderate relationship between the number of modalities used during a lesson and student engagement increased that was marginally correlated with student engagement (r=.3, p<.05). That is, lessons incorporating a greater number of modalities had greater student engagement (Table F-8). Table F-7 FLES Classroom Observation Cross-tabs of Implementation Level by Student Engagement Engagement Very Low (0%) Low (40%) Moderate (60%) High (80%) Full () Total Teacher Beliefs and Attitudes Low (5 or fewer elements) 0 0% 1 50% 1 33% 1 17% 4 11% 7 15% Implementation Level Moderate (6-7 elements) 0 0% 1 50% 67% 5 83% 14 40% 48% Full (8 elements) 0 0% 0 0% 0 0% 0 0% 17 49% 17 37% Total 0 0% 3 6 35 46 When FLES and classroom teachers were asked to report on their expectations for student performance, both groups reported similarly high expectations (approximately five on a six-point scale; Table F-9). Classroom and FLES teachers beliefs and attitudes were also compared for novice vs. experienced teachers, and no differences were found with respect to alignment of FLES and POS instruction. Interestingly, novice and experienced FLES teachers did not differ with respect to their teaching efficacy for FLES, indicating that even experienced teachers described middling levels of confidence in their ability to teach FLES and core content POS effectively. When teachers from schools instructing in western vs. non-western schools were compared, some differences were noted. Independent samples of t-tests indicated that expectations for students were significantly (p<.05) higher among FLES teachers of western (M=5.31, sd=.47) than non-western (M= F-6

4.85, sd=.59) languages. In contrast, teacher efficacy was significantly higher (p<.05) among FLES teachers of non-western (M=4.1, sd=.58; western: M=3.7, sd=.34) languages. Table F-8 Student Engagement by Number of Modalities Used During FLES Lesson Student Engagement 0% 40% 60% 80% Three Modalities 0 0% 0 0% 1 10% 1 8% 6% Four Modalities 0 0% 7 70% 4 33% 10 9% Five Modalities 0 0% 0 0% 0% 7 58% 3 66% Total 0 0% % 10 1 35 Table F-9 Descriptives for FLES and Classroom Teacher Expectations for Student Performance Collaborating FLES Teachers (n=8) Classroom Teachers (n=38) Mean Standard Mean Standard Deviation Deviation 5.17 0.57 5.6 0.86 FLES program staff indicated that collaboration between FLES and regular classroom teachers is critical to producing effective FLES lessons that integrate both foreign language and core content. Towards that end, program staff expect FLES teachers to participate in grade-level team meetings in order to understand what core content is currently being covered by the classroom teacher and what lessons to select from the FLES POS. Program staff also felt that it was essential for FLES and classroom teachers to communicate and cooperate about individual student issues, such as behavior management for Special Education students. Quantitative survey data indicated that while FLES and classroom teachers held similar average perceptions regarding their collaboration with one another, the two groups of teachers differed in the range of scores with classroom teachers spanning from Disagree to Agree/Strongly Agree, and FLES teachers more highly skewed and spanning only Somewhat Disagree/Somewhat Agree to Strongly Agree (Table F-10). That is, the classroom teachers represented a far greater range of opinions regarding their collaboration with the FLES teacher (sd=.94), than did the FLES teachers (sd=.48). Mean comparisons between novice and experienced teachers, and teachers at schools instructing in western vs. non-western comparisons, yielded no significant differences. On the qualitative data, which indicated classroom teachers evenly split between positive and negative perceptions of collaboration with the FLES teacher, most teachers with positive perceptions praised the FLES teachers attendance and participation in grade-level planning meetings as a basis for excellent collaboration. In contrast, most teachers with negative perceptions indicated that the FLES teachers had not attended these meetings. Thus, involvement in the grade-level meetings as a means of facilitating F-7

collaboration between FLES and classroom teachers appears to be uneven. However, even among classroom teachers who noted negative perceptions, there was an underlying belief in the critical nature of this collaboration for program success. Table F-10 Descriptives for Collaboration by FLES and Classroom Teachers Minimum Maximum Mean Standard Deviation FLES Teachers (n=8) 3.57 5.57 4.66 0.48 Classroom Teachers (n=38) 1.86 5.43 4.49 0.94 Another indicator that collaboration between FLES and classroom teachers may need improving were suggestions that future professional development activities should focus on building relationship and collaboration skills. Among classroom teachers providing suggestions for program-related professional development that would improve Fidelity of Implementation or Outcomes, the largest number (n=6) referenced the need to dedicate professional development time to improving collaboration skills. In addition, two FLES teachers made similar recommendations. Neither school-based administrators nor FLES program staff indicated a need for this type of training, which presumably means that the concerns over collaboration between FLES and classroom teachers has been severe enough to be of great concern to management. In summary, while there do not seem to be huge concerns about site-level implementation fidelity for collaboration between FLES teachers and classroom teachers, there is evidence that classroom teachers are not consistently satisfied with the quality of the collaboration. Since this activity is considered essential to core content integration, additional attention may need to be paid to have greater consistency in this activity from site to site. In particular, without specific benchmarks to guide both FLES and classroom teachers in the characteristics of quality collaboration, it is difficult for teachers to aim for and recognize effective collaboration. Component Analysis, Judgment and Rating Prerequisites Site implementation of program activities reflects categories of surface features as described in the site-level model of the program s definition. Activities defined in the site-level model reach expected audience. Standard Elements 1. Site implementation of program activities is aligned with the overall purpose of the program in support of the specific linked outcomes defined in the site-level model of the program s definition. 5. Program is purposefully implemented at sites with a clear sense of the program s intended scope and reach and the application of this scope and reach by sites. For the Fidelity of Implementation component, all evidence indicates that the program easily meets the Prerequisites of quality. Schools are regularly engaged in the four core activities (Lesson delivery, Contributions to Blackboard, Collaboration, Participation in professional development) with moderate to high fidelity, and the program is reaching the students it is designed to serve. 5 As identified by program staff during the exploratory evaluation. F-8

Moving on to the Standard Elements, Standard 1 is fully met. That is, schools have generally aligned their FLES instruction with the expectations set by the program office. This description is, however, tempered by the fact that several of the activities require enhanced Program Definition (such as defining benchmarks of quality implementation), which could impact these implementation levels in the future. In other words, the schools are, by and large, faithfully implementing what the program has set out for them to do, but there is a need for the program to define specific expectations for some of the activities to minimize the variation that is occurring at schools in some activities. Standard proves more problematic for the FLES program as some of the activities schools are having the greatest difficulty implementing with fidelity (e.g., joint planning time) are critical to meeting program goals (e.g., achievement of students in core content areas). This may be related to improving school-based staffs understanding of the FLES program model, especially how individual activities and their implementation characteristics are intended to achieve expected outcomes. Nonetheless, the evaluation evidence indicates that multiple schools are making implementation choices (e.g., little or no opportunity for planning between FLES and classroom teachers, separate FLES classroom, scheduling planning time so teachers cannot remain in the classroom with students, scheduling no transition time between 30-minute lesson blocks, etc.) that do not adhere faithfully to the program s intentions. Evaluation Question and Response To what extent do participating sites implement the program as defined by central office? (Evaluation Question ) Overall, FLES schools implement the four activities as defined by program staff with moderate to high fidelity. Evidence indicates that the basic expectations for FLES lessons routinely appear in grade 1 and classrooms. Even the characteristic with the lowest frequency of appearance, teachers actively observing the FLES lesson, appear in the majority of FLES lessons. Other lesson characteristics, such as integrating core content and foreign language POS, also appear routinely. On the remaining three implementation activities, the FLES program is also fairing well. The vast majority of FLES teachers contribute to the FLES Blackboard site every one to two weeks, which is at or near the expectation that teachers post lessons weekly. Teachers also routinely collaborate with classroom teachers and attend program-specific professional development as expected by the program. Thus, although more clearly-defined expectations are warranted for activities, as currently defined by central office, schools are implementing site-based program activities with fidelity to the program s definition. F-9

APPENDIX G FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL ADDITIONAL DETAILS ABOUT OUTCOMES

Additional Evaluation Evidence Foreign Language in the Elementary School Additional Details about Outcomes The following description provides the perspective of principals, classroom teachers, and Foreign Language in the Elementary Schools (FLES) teachers regarding the student outcomes they perceive as demonstrated in relation to FLES implementation. The evidence used to support the principal and teacher observations of outcomes should not be considered objective evidence of program impact. However, it is important to note the outcomes that these stakeholders believe are stemming from the program and the specific subgroups of students they believe are benefitting the most. Perceived Outcomes Principals All outcomes reported by the administrators conform to the program s model. The most frequently mentioned outcome by administrators was foreign language acquisition with 17 of administrators indicating they saw this as an outcome of students participation in FLES. Approximately one-third of the administrators also reported core content learning (n=6) to be an outcome of FLES instruction. Four administrators indicated that they were unsure of the outcomes at that point. With respect to the evidence, administrators cited when indicating the impacts of the FLES program on students, the most frequent source cited was demonstrations by students outside class (n=5). In a few cases, the evidence cited was more about students exposure to learning opportunities (e.g., international nights, production of news shows or other media, teachers pulling up information on the language) than evidence of student outcomes. Classroom Teachers All outcomes reported by classroom teachers in response to this question conform to the program s model. The outcome mentioned most frequently by classroom teachers was foreign language acquisition with 5 of 48 classroom teachers indicating they saw this as an outcome of students participation in FLES (it should be noted that three additional classroom teachers provided answers to this question that focused on the instruction in foreign language, but did not specifically indicate the students learning or acquisition of foreign language; so they are not counted here). Approximately one-third of the classroom teachers also saw cultural knowledge or understanding (n=14) to be an outcome of FLES instruction. While approximately half of the classroom teachers (n=3) mentioned core content in relation to this question, none specifically attributed core content learning of students to FLES instruction. Teachers indicated that the students were using vocabulary in the target language during regular core content instruction (n=8), but none of the teachers went so far as to say that students were learning core content from FLES. More frequently, the classroom teachers commented positively on the integration of core content into the FLES lesson or saw FLES instruction as a review of concepts previously taught in English (n=1), including specific mentions of integrating mathematics (n=5), science (n=5) and social studies (n=4). A few classroom teachers viewed FLES instruction as taking away classroom time from core content instruction (n=3). While comments were generally positive, much of this falls more into the activities of the program rather than its outcomes. These findings suggest an overall perception by the classroom teachers that FLES is not a strong contributor to students core content learning. Classroom teachers did not cite any other outcomes beyond these three critical student outcomes in response to this question, thus aligning all their responses with the main American Council for Teaching Foreign Language (ACTFL) standards of interest in this evaluation. With respect to the evidence cited by classroom teachers when responding to this question, most either did not provide specific support or G-1

indicated that they saw students demonstrate productive or receptive language skills, such as understanding the FLES teachers directions or using vocabulary outside the FLES lesson. FLES Teachers All outcomes reported by FLES teachers also conform to the program s model. Interestingly, in contrast to principals and classroom teachers, FLES teachers most frequently mentioned cultural awareness (n=4) as an outcome of participating in FLES. FLES teachers also frequently mentioned foreign language achievement (n=18) and core content knowledge (n=17) as stemming from FLES participation, and at approximately equal rates. Unlike the classroom teachers, fewer FLES teachers delved into specific core content areas with only a few mentions of language arts (n=3) and mathematics (n=). Several FLES teachers went beyond these three main outcomes to highlight fearlessness of language (n=7) and flexible thinking (n=6) as student outcomes. Subgroup Benefits Classroom Teachers Fifteen of the teacher respondents indicated their belief that all students benefit from participation in FLES. Some of these respondents cited different benefits for different groups (e.g., self-esteem for native speakers, repetition of content for Special Education students, enrichment for Gifted and Talented (GT) students) but felt that the FLES program was a benefit to all the students in their classrooms. This was the most common response among classroom teachers. Among teachers who saw specific benefits to certain subgroups of students, limited English proficiency (LEP) (n=9), native speakers of the target language (n=7), and high achievers (n=5), were most frequently referenced. Five classroom teachers also specifically referenced Special Education students as a subgroup that did not fair as well as most students with FLES instruction. FLES Teachers In contrast to classroom teachers, the most frequent reference to subgroups by FLES teachers was regarding benefits to Special Education students (n=18). Although a couple of teachers (n=3) referenced Special Education students as having a more difficult time with FLES instruction, most FLES teachers cited the benefits to this subgroup of students. This indicates that the classroom and FLES teachers perceive the instruction of Special Education students differently from one another. Additional groups that FLES teachers saw benefitting from FLES instruction aligned more closely with the responses of classroom teachers: FLES teachers perceived LEP (n=11) and native speakers (n=6) of the target language to be gaining the most from FLES instruction. Component Analysis, Judgment and Rating The judgment and rating of FLES Outcomes are based on the program s success in meeting the following Prerequisites and Standard Elements: Prerequisites Critical outcomes 6 have made progress at this point in the program model. 6 OPE identifies critical outcomes within the Outcome evaluation questions. For the FLES program, critical outcomes are the Communication, Cultures, and Connections standards described by ACTFL (American Council on the Teaching of Foreign Languages), as well as differential outcomes for LEP (Limited English Proficient) or Special Education students. G-

Standard Elements 1. There is evidence that critical outcomes can be reasonably attributed to program activities as measured by Fidelity of Implementation.. Critical outcomes meet expectations as defined at this point in the program model. Because the FLES evaluation design calls for student outcome data to be collected starting in Year Two, the Year One Interim Evaluation Report does not include findings, discussion, judgments, or ratings for this program component or its associated evaluation questions. A rating for the Outcomes component will be provided in the Year Two Evaluation Report. Evaluation Questions and Responses To what extent do students participating in FLES meet the selected standards described by the American Council on the Teaching of Foreign Language (i.e., Communication, Cultures, and Connections)? (Evaluation Question 3a) See note above in the Component Analysis section for information regarding the omission of the response to this evaluation question. A response to this question will be included in the Year Two Evaluation Report. Do some groups of students demonstrate stronger achievement benefits from FLES than others (i.e., LEP, Special Education)? (Evaluation Question 3b) See note above in the Component Analysis section for information regarding the omission of the response to this evaluation question. A response to this question will be included in the Year Two Evaluation Report. G-3

APPENDIX H FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL ADDITIONAL DETAILS ABOUT PROGRAM COSTS

Foreign Language in the Elementary School Additional Details about Program Costs This appendix provides additional information collected during the course of the evaluation to inform judgments about the program quality components including detailed information about program performance on each Prerequisite and Standard Element, and provides evidence and judgments about the Program Costs component evaluation question. Additional Evaluation Evidence The Office of Program Evaluation s (OPE) approach to gathering evidence to describe Program Costs includes program budget, expenditures, and additional funding sources. This approach differs from how Program Costs are calculated for Foreign Language in the Elementary School (FLES) program in the Program Budget Book. As can be seen in the left-hand columns of Table H-1, the Program Budget Book includes salary and benefits for 8.5 teachers and one full-time resource teacher at 9 schools. The Program Budget Book did not allocate funds to program operations in 009, so none of the additional funds required to implement the program are included in the costs. To provide a full understanding of the costs of administering the program, the Year One Interim Evaluation Report details costs beyond the Program Budget Book. As can be seen on the right-hand side of Table H-1, the description of costs for the Year One Interim Evaluation Report includes three additional sources of funds, World Languages Office support, the Department of Education s Foreign Language Assistance Program (FLAP) grant support, and congressional earmark. Adding these sources of funding provides a better sense of the true cost of ownership. Program Costs include salary and benefits for 31 teachers (8 positions funded by school-based support, two positions funded by the FLAP grants, and one position funded by the congressional earmark) at 3 schools, as well as the full-time resource teacher. It also includes staffing costs for members of the World Language section who devoted effort to FLES (coordinator and educational specialist at 0 percent effort each, and the administrative assistant at 10 percent effort). The World Languages section devoted an additional $930 of its resources to provide professional development funds for teachers to attend regional foreign language conferences and $8,381 to pay for substitute teachers so that FLES teachers could attend professional development meetings ($68) and, in addition, to pay for teacher time during the summer to revise the FLES Program of Studies (POS), $,914. The total costs of operating the FLES program are estimated to be $,916,301. Table H-1 also presents the Cost per Participant (CPP) for the FLES program. The program sites costs as $171 per student based on the assumption that all students at a school are receiving exposure to the target language either through direct instruction (early grades) or through visits to their specials, signage in the school, and school-wide cultural events (upper grades). Since the program defines itself as offering instruction, rather than just exposure, the evaluation estimates CPP based on the number of students who actually received FLES instruction, 7,63 in FY 009, for a cost of $383 per participant. The $383 figure includes all funds expended by the program, regardless of the source. This value would be only slightly lower, at $377, if the $43,000 in external funding that supported materials, equipment, and staff development not available to other schools were removed from the CPP calculation. That is, the funding available to externally funded schools for extras such as materials amounts to approximately six dollars per student across the entire program. The cost of operating FLES was compared to the cost of operating Partial Immersion, the only other program in FCPS whose goal is to develop communicative competence in a world language. As with the H-1

FLES program, the costs of Partial Immersion used in the evaluation are based on the costs of operating the program in elementary schools which differs from the costs in the Program Budget Book (Table H-). The differences in costs stem from the Program Budget Book including teacher positions at the middle and high school levels, including partial effort from one specialist in the World Language section (rather than all support from World Language section staff), and not including additional funds used by the World Languages section to support the program. The rest of this section details the costs of the Partial Immersion used for the evaluation. Table H-1 Comparison of Elements Used to Define Program Costs * FCPS Program Budget Book FY 009 Approved School- Based Support Non- School- Based Support Program Costs Year One Interim Evaluation Report FY 009 Non- World School- Language FLAP Based s Office Grants Support Support Support School- Based Support H- Congressional Earmark Full-Time Salaries $1,789,358 $69,45 $1,77,736 $85,589 $45,455 $16,63 $63,31 Part-Time Salaries 0 0 0 0 0 0 0 Employee Benefits 66,06 5,687 655,91 31,668 16,819 46,851 3,45 Operating Expenses 0 0 0 0 0 0 0 Materials & Supplies 0 0 0 0 0 10,000 9,163 Equipment 0 0 0 0 0 0 15,75 Staff Development 0 0 0 0 930 60 0 Other 3,54 8,381 Sub-totals $,48,648 $95,11 $,48,648 $117,57 $66,746 $19,475 $111,175 Total $,546,53 $,916,301 Number of Schools 9 3 Number of Students 14877 763 Cost per Participant (CPP) $171 $383 *Due to rounding of values, totals and sub-totals may not sum exactly. Program staffing funds for the Partial Immersion program are estimated differently than for FLES. For FLES, teachers are in addition to the regular classroom teachers. In Partial Immersion, the regular classroom teacher delivers the instruction in the target language during mathematics, science and health. Attrition from the Partial Immersion program occurs in the upper grades; consequently, teachers need to be hired to teach the non-immersion classes. At the elementary level, this typically results in the allocation of a minimum 1.0 position to balance the lower enrollment in the upper grades of the Immersion classes with the non-immersion classes. Based on the documents provided by the program manager in fiscal year 009, a total of 15 teacher positions were funded for non-immersion teachers in Immersion schools. In addition, the 13 Immersion schools are allocated a full-time instructional assistant to aid in the classroom. The program costs reported in Table H- are based on the average teacher and instructional assistant salary and 37 percent for benefits. The Immersion Program also gives each school $500 a year to purchase supplies and $387 to support professional development for staff. The column, Non-School-Based Support, describes the funds required to pay the program staff that operate the program. While the Program Budget Book only includes funds for administrative support, the

program is actually supported by additional central office staff. The program manager provided estimates of the amount of time World Languages staff spend supporting the program. The table includes 10% effort for the World Languages coordinator, 80% effort for the educational specialist, and 50% effort for the administrative assistant to support the lottery. In addition to school and non-school-based support funding, the Immersion Program relies on supplemental funding from the World Languages Office. The program manager expended $1,185 to send teachers to local foreign language conferences. The office also spends funds to pay for substitute teachers so Immersion teachers can be trained in and administer the foreign language assessment tool ($1,564). In addition, the office expends funds to translate instructional materials from English into the target language ($9,34.80). The total costs of operating the Partial Immersion program are estimated to be $1,994,404.10. Table H- Immersion Program Cost Description * FCPS Program Budget Book FY 009 Approved School- Based Support Non- School- Based Support H-3 Program Costs Year One Interim Evaluation Report FY 009 School- Based Support Non- School- Based Support World Languages Office Support Full-Time Salaries $1,499,85 $34,71 $1,316,43 $ 113,77 0 Part-Time Salaries 0 0 0 0 0 Employee Benefits $554,945 $1,843 $487,069 $4,078 Operating Expenses 3,947 0 0 0 0 Materials & Supplies 0 0 $6,500 0 0 Equipment 0 0 0 0 Staff Development 0 0 $5,031 0 $1,185 Other $,389 Sub-totals $,078745 $47,555 $1,815,03 $155,807 $3,574 Total $,16,300 $1,994,404 Number of Schools 8 13 Number of Students 3,99 390 Cost per Participant (CPP) $606 *Due to rounding of values, totals and sub-totals may not sum exactly. Table H- also presents the CPP for the Partial Immersion Program based on the figures used for the evaluation. The total cost of the program was divided by the number of participants, based on service contacts in elementary school. The CPP for Partial Immersion is $606 per participant. Component Analysis, Judgment and Rating Prerequisites Program spending is linked to activities that are intended to support program objectives. Program staff can provide sufficient information to calculate the CPP. A rating of cost-effectiveness (CE) is available.

Standard Elements Required to meet Benchmark 1. Program is at least as cost-effective as the average of programs of similar type (supplementary, alternative) serving similar target populations (general education, special education, adult). In addition to Standard 1, Standard is required to rate Above Benchmark.. Program routinely reflects on steps to reduce or contain costs in order to optimize outputs (Cost Optimization Level = High). The Program Costs component requires that Outcomes data be available to assign a rating. The Year One Interim Evaluation Report does not include FLES Outcomes data because it will not be available until the second year of the evaluation when sufficient numbers of students have reached the first assessment point in Grade 3. Response to Evaluation Question To what extent does the FLES program maximize cost-effectiveness, particularly in comparison to elementary Immersion programs? Although it was not possible to rate the program on Program Costs, information was collected to report on the extent to which the FLES program attempts to optimize costs and how its efforts compare to the Partial Immersion program. The FLES program spends the majority of its funds, 9 percent, to support the teachers who provide instruction in the target language. The remaining eight percent of funds are used to support teachers instruction through ongoing professional development, revision of the FLES POS, and funding for program staff to support teachers delivery of instruction. The program maximizes the value of attending conferences by having program staff and teachers who attend conferences provide turnaround training to their peers. In addition, the program offers professional development through monthly meetings, two inservices, and peer observations. The program optimizes its costs, in part, by ensuring that program funds support the most critical aspects of the program. The other way in which the program has optimized costs is through grants. The program was fully funded in four schools during FY 009 (three schools had federal grants [Foreign Language Assistance Program (FLAP)], and one had a congressional earmark). The grants and earmark defray the initial startup costs and provide generous support for materials, special activities like summer camps, and equipment costs. The grants limit the languages that can be taught to critical languages. The program does not anticipate seeking further grants once the FLAP grants expire in FY 010. The one area in which the program is the weakest in optimizing costs is the staffing formula. The program provides a.5 position in a school the first year a language is taught in grade 1 and possibly Grade. The position is augmented to full-time in the second year with additional grade levels added each year. Once a teacher reaches 400-450 students, she or he could be teaching approximately 15 classes, across five to six grades, two times a week. The formula is based on the assumption that once teachers have a full instructional load, they will have very little time to plan their lessons, so generous time is allocated in the initial year to provide planning time. In the first year, a teacher is likely to teach about 6 hours a week (3 classes, twice a week for 30 minutes); the remainder of his or her time is allocated to developing lessons for the current and higher grade levels; posting to Blackboard; attending grade-level planning meetings and FLES meetings; and biannual peer observations. Assuming 0 percent of a teacher s time is allocated to lesson planning, the program is paying approximately $500,000 of its funds to this purpose. This is an extensive amount of H-4

money given the concerns teachers voiced about the Blackboard website (see Appendix F, Additional Details about Fidelity of Implementation). Without having Outcomes data for FLES, it is impossible to compare its cost-effectiveness to the Partial Immersion program. Based on the cost description, Partial Immersion is more expensive, $606, vs. $383 for FLES; however, since more time is spent in the target language, students are likely to develop higher levels of proficiency. It should be noted that the attrition rate from Partial Immersion is high, so fewer students will develop proficiency than in FLES, where all students are exposed to the program. A judgment about the relative cost-effectiveness of FLES compared to Partial Immersion will be made in the Year Two Interim Evaluation Report. H-5

APPENDIX I FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL SUPPLEMENTAL PROGRAM COMPONENTS (MONITORING SYSTEM, PROFESSIONAL DEVELOPMENT, SITE IDENTIFICATION)

Monitoring Systems Evaluation Evidence Foreign Language in the Elementary School Supplemental Program Components (Monitoring System, Professional Development, Site Identification) Evaluation evidence for this component was gathered from three sources: (a) program staff, (b) schoolbased administrators, and (c) program documents. Program staff descriptions of their monitoring system indicate that they rely heavily on site visits to monitor the program s implementation. Both the description provided by Foreign Language in the Elementary Schools (FLES) program staff and program documents provided by them, indicate that feedback is provided to FLES teachers regarding the visit. The monitoring focuses on aspects of instruction that the program has defined as expectations for schools. For example, program staff reported from the program s notes of the visit support, they look for indicators of good collaboration between FLES and classroom teachers, such as the regular classroom teacher s presence in the room during the lesson, and active observation including smiling and some responses in the target language, etc. In addition, the program routinely monitors the Blackboard site for lesson posting and records this information in a monitoring spreadsheet. In addition, FLES staff reported that they monitor FLES teacher understanding at the monthly FLES meetings for adherence to program expectations. Lastly, program staff gathers FLES teacher schedules and looks for grade-level meetings as part of the schedule, which was also supported by program records supplied to the Office of Program Evaluation (OPE). If necessary, one-on-one feedback is given to the FLES teacher; if unsuccessful in correcting implementation concerns, program staff makes contact with administrators at the school. Emails and responses from principals corroborated the collaboration between principals and program staff on implementing with fidelity. No evidence was found to support that FLES monitors program-level activities or outcomes. Component Analysis, Judgment and Rating Prerequisites Program can identify available monitoring information about some of the program s inputs defined in the model. Program can identify available monitoring information about some program activities defined at the program-level (process) or some program activities defined at the site-level (site implementation). Program can identify available monitoring information about some program outcomes defined at the program-level or some program outcomes defined at the site-level. Program periodically reflects on monitoring information about past performance (either process/site implementation or outcomes). Standard Elements 1. Program employs a self-directed and sustained set of procedures to routinely and systematically gather monitoring information about all critical activities and outcomes at both the program- and site-levels.. Program and/or program sites interpret monitoring information in relation to set standards (e.g., benchmarks, criteria) that reflect specific objectives for each of the program s critical activities and outcomes. 3. Program and/or program sites routinely engage in an active process of reflection and making sense of past performance for the express purpose of orienting the program toward modifications to improve future performance. 4. The program uses monitoring information to support Plan, Do, State, Act (PDSA) cycles of continuous improvement. I-1

Judgment: The FLES program has done a good job of implementing self-directed procedures to monitor some aspects of the activities and outcomes highlighted in the program s definition. However, the lack of defined benchmarks for program activities hinders effective monitoring of all critical site-level activities. As has been previously described in relation to the Program Definition and Fidelity of Implementation components, the program lacks benchmarks for activities such as collaboration between FLES and classroom teachers. It also lacks standards for student outcomes such as foreign language during nonassessment years, core content achievement, and cultural awareness. Thus, while the program has in place strong monitoring for some site-level activities and program outcomes and clearly reflects on the information (meeting all Prerequisites), additional work needs to be done to meet any of the Standard Elements that speak to a more comprehensive monitoring system with benchmarks and Standards. Rating: (Basic) Professional Development Evaluation Evidence The Office of Program Evaluation (OPE) used multiple sources to gather evidence about professional development. These included survey responses from FLES teachers, regular classroom teachers, and principals, as well as interview data from FLES program staff. In addition, program documents provided by the World Languages staff helped corroborate and expand on the professional development activities engaged in during SY 008-09. Evidence from all sources showed that professional learning opportunities for FLES teachers, rather than classroom teachers, were emphasized by the program. This is in keeping with the responsibilities taken on by the FLES program staff, as described to OPE at the inception of the evaluation. The FLES program staffs (who are part of the Office of World Languages staff) indicated (and program documents corroborate) that they provide professional development opportunities for FLES teachers by engaging in the following tasks: Designing, preparing and implementing monthly staff development meetings; Designing, preparing and implementing two county-wide in-services a year; Identifying and advertising conferences and workshops relevant to FLES teachers; and Providing individualized support, as needed, by FLES teachers. Furthermore, program staff held expectations that FLES teachers participate in the following program-specific professional development: Attending monthly professional development meetings; Attending two countywide in-services (August and January); and Participating in peer observations and submitting feedback. Of these activities, program staff felt that the most critical for FLES teachers was (a) monthly meetings, which allowed for turnaround training and opportunities for FLES teachers, who typically work alone, to connect with one another, and (b) peer observations, which facilitated discussions about lesson delivery among FLES teachers. The FLES teachers listed the monthly meetings (n=19) and conference attendance (n=17) as the most frequent professional development activities. So, while peer observations were a required professional development activity during SY 008-09, few FLES teachers (n=) referenced it when discussing professional development. On the FLES teacher survey, 79 percent of FLES teachers indicated that they were either satisfied or highly satisfied with the FLES-related professional learning opportunities in which they had participated during SY 008-09; with all responses averaging 4.89 (sd=1.34) on a six-point scale (1=Highly I-

Dissatisfied, = Dissatisfied, 3=Somewhat Dissatisfied, 4=Somewhat Satisfied, 5= Satisfied, 6=Strongly Satisfied (Table I-1). Follow-up analyses indicated no difference in satisfaction between novice and experienced teachers. When asked specifically about the impact that professional development had on their knowledge and instructional practice, FLES teachers perceived primarily positive results from participating in professional learning opportunities. The average response on most items was five on a six-point scale (1=Strongly Disagree, =Disagree, 3=Somewhat Disagree, 4=Somewhat Agree, 5=Agree, 6=Strongly Agree), including those referring to professional development increasing understanding of the FLES curriculum, improving FLES instruction, improving the integration of core content into FLES lessons, and teaching effectively. Only one item, addressing professional developments impact on FLES teachers ability to collaborate with regular classroom teachers, yielded a slightly lower mean response (4.5). Survey Item Table I-1 Descriptive Statistics for FLES Teacher Survey Responses about Professional Development (n=8) Overall, how satisfied were you with the FLESrelated professional learning opportunities in which you participated this year? Mean Standard Deviation 4.89 1.34 During this year, FLES-related professional learning opportunities in which I participated... Increased my understanding of the FLES curriculum. 5.00 1.36 Helped me improve my FLES instruction in first and second grade classrooms. Helped me improve my ability to integrate core content standards into my first and second grade FLES lessons. Gave me the support I needed to teach effectively in first and second grade classrooms. Helped me improve my collaboration with first and second grade classroom teachers. 5.00 1.39 5.04 1.37 4.93 1.36 4.54 1.37 Principals and teachers were asked to suggest possible future professional development. Overall, administrators did not have many suggestions regarding additional professional development that could be offered to improve program implementation or impact. Four of administrators replied that no additional professional development was needed, while an additional two administrators indicated they were unsure what might be beneficial. Among the remaining 16 administrators, six suggested informational sessions for classroom teachers about to begin FLES in a subsequent grade, or others new to FLES, and five administrators suggested language classes for the classroom teachers. The 8 FLES teachers offered more suggestions, but no more than three coalesced around a single idea, resulting in a large number of possible topics. Topics primarily focused around trainings for the FLES teachers, but some reflected trainings for other stakeholders. Classroom teachers most frequently responded that they were uncertain or that they had no suggestions (n=13). Six classroom teachers believed that professional learning that either focused on improving their collaboration or allowed them more time to collaborate would be the most useful (n=6). For themselves, a few classroom teachers suggested they receive professional development about the FLES program to help them understand it better (n=4) or how to support the FLES program better within their classroom (n=4). Some classroom teachers also suggested that FLES teachers receive professional development to improve their instruction, including their I-3

knowledge of Fairfax County Public Schools Program of Studies (POS) (n=3), how to differentiate (n=3), classroom management (n=), active learning (n=), and grouping practices (n=1). Lastly, two classroom teachers believed that parents/community should receive training or information on the FLES program. Component Analysis, Judgment and Rating Prerequisites Program offers staff professional development aligned with surface features of program s defined activities. Staff members participate in assigned professional development activities. Standard Elements 1. Program engages program staff in professional development addressing the unique training needs of the program.. Program staff provides a data-driven or research-based rationale for the proposed level of staff s participation in professional development activities. 3. Program engages program staff in professional development aligned with improving specific program outcomes. 4. Program staff provides embedded professional development to help staff members implement new skills and knowledge. Judgment: The evaluation evidence supports the FLES program s involvement in providing FLES teachers with the specific program support they need to implement FLES at the schools and FLES teachers participation in these professional development activities. Thus, the program fully meets all Prerequisites and Standard Element 1, earning a benchmark rating of 3 on this program quality component. Rating: 3 (Benchmark) Site Identification Evaluation Evidence Evaluation evidence for this component was gathered from three sources: (a) cluster assistant superintendents, (b) program staff, and (c) program documents. Program staff and program documents, including an application form, indicated a set procedure for identifying FLES sites. The process for applying to become a FLES school begins with the assistant superintendent of the Department of Instructional Services (ISD) sending out a memo in the fall inviting elementary schools to apply to implement the FLES program. [No memo was sent out in fall 008 since there was no budget to expand the FLES program at that time.] The application process then includes the following steps: Ask cluster office for approval to apply; Program staff go out to the school to explain the program to the teachers; Program staff go out to the school to explain the program to the community; School conducts a survey to gauge support for the program and gathers feedback on language preferences; Schools are required to collaborate with the MS/HS principals regarding language selection; Schools submit their applications to their clusters for signature; World Languages section (WL) coordinator collates all the submitted applications; WL coordinator goes to Academic Learning Group (ALG) with list of schools that applied; ALG discussions focus around equity in elementary foreign language offerings by cluster; and Cluster assistant superintendent (CAS) comes to an agreement on the number of programs in each cluster, but the CAS has say over which specific school with his or her own cluster receives FLES. I-4

In general, the seven clusters that contributed responses about FLES site identification did not provide many suggestions regarding improvements that might be made to the FLES site selection process. In fact, two of the clusters specifically indicated that the current process has worked very well, and others indicated the process should not be changed. Four of the seven clusters praised the involvement of the community and parents in the process or stressed the critical need to involve this stakeholder group. In addition, two clusters specifically indicated their opposition to continued FLES expansion given current budget constraints. With respect to improving the site identification process, only a few suggestions were offered by clusters. The language selection that accompanies a school being identified to participate in the FLES program was stressed by two clusters that saw a need to align languages with middle school and high school offerings, including with multiple pyramids when the elementary school is a split feeder. Another cluster recommended looking at community involvement. One cluster also suggested a stronger emphasis on schools understanding the selection process, implying a need to communicate the selection criteria more clearly than has been done to date. Component Analysis, Judgment and Rating Prerequisites There is a broad description of criteria for identifying sites. The program applies these criteria in a consistent manner when identifying sites. The program provides information to sites that clarifies how they fit the criteria. Standard Elements 1. There are specific criteria for identifying sites that are documented, clear, objective, valid, and relevant.. The criteria for identifying sites are aligned with the overall purpose of the program. 3. The program is flexible in making adjustments to attempts at recruiting sites (including expansion). 4. The program collaborates with potential sites in identification process. 5. Sites coordinate the program with other activities at the site based on the contribution of the program to expected outcomes. Judgment: FLES documents clearly indicate that the program has a specific application process available to use whenever expansion is desired. The process was developed several years ago and used with the schools that were added to the FLES program during the last two school years. The identification process criteria, as represented on the application memo and form, indicates the basis for the decisions that are made. There is some evidence from clusters that more clarity in the process would be helpful to schools. The program has incorporated collaboration with potential sites into the process, including sending program staff out to potential FLES schools and their communities to explain the program. Thus, the program fully meets all Prerequisites and Standard Element 4, earning a benchmark rating of 3 on this program quality component. Rating: 3 (Benchmark) I-5

APPENDIX J FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL METHODOLOGY AND INSTRUMENTS

Foreign Language in the Elementary School Methodology and Instruments This appendix provides information about the Office of Program Evaluation s (OPE) approach to conducting evaluations, as well as detailed information about the methodology used to collect data for the evaluation report. Evaluation Approach The Office of Program Evaluation (OPE) has a team approach to designing evaluations. Teams generally comprise eight to ten people including, the lead- and co-evaluator, director of OPE, program manager, and additional central office and school-based personnel selected to represent a variety of stakeholder perspectives about the program. The purpose of the team is to provide input at critical stages during the evaluation. See Table J-1 for a description of the stages and team member roles. Team members are not necessarily expected to participate in each stage. Requests for input are based on team member expertise. Stage Design Instrumentation Data Collection Data Analysis and Synthesis Report Writing and Presentation Table J-1 Description of Evaluation Team Roles Evaluation Team Member Role Provide information regarding informational needs, inclusion of box score components into design, feedback on draft evaluation questions, potential data sources, and timelines for data collection. Provide feedback on the scope and reasonableness of evaluation instruments (surveys, interview/focus group protocols, additional student measures when appropriate). Provide feedback on timing and communication mechanisms used to collect data using selected instruments. Provide feedback on the reasonableness of the conclusions based on data, particularly in cases of mixed results. Provide feedback on written reports and presentations, with particular sensitivity to clarity of language, utility of the information in meeting informational needs, and sensitizing OPE to any political hot spots. The evaluation team members for Foreign Language in the Elementary School (FLES) are as follows: Marty Smith, Cluster Director, Cluster I Tonya Cox, Principal, Gunston Elementary School Steve Gossin, Principal, Cherry Run Elementary School Paula Patrick, Foreign Languages Coordinator, Instructional Services Department Robert Harrison, IBMYP Coordinator, Glasgow Middle School Marie Hinton, FLES Resource Teacher, Instructional Services Department Evaluation Data Sources The FLES Evaluation Design (Appendix D) describes data collection sources and timelines for FLES. Below is a list of data sources. The rest of this document will describe the methodology used to collect data from each of the following data sources: Classroom observations FLES Teachers Classroom teachers at FLES schools Principals at FLES schools Cluster Offices FLES Program Staff Program artifacts and documents FLES Lesson Plans ISD Budget Liaison J-1

The focus of the Year One Interim Evaluation Report is to examine FLES implementation, particularly in grades one and two where the vast majority, 69 percent, of FLES students were located during SY 008-09 7. Of the 3 schools operating FLES, two do not offer world language instruction until Grade 3, leaving 30 schools for the evaluation. 8 Specifically, Daniel s Run begins teaching Latin in grade 3 and Mantua starts teaching Italian in grade 3. Neither of these schools was targeted for data collection efforts in the first year of the evaluation because their implementation model is not aligned to the FLES model. Future evaluation reports will examine program outcomes once a sufficient proportion of schools implementing the FLES model have students in grade 3, the first year that language proficiency is assessed. Currently, only seven of the 3 FLES schools have instruction in grade 3 and of those, only five have students who have had consistent instruction since grade 1. Some of these schools began implementing FLES before the current program model was developed so judgments about the impacts of the program might not be generalizable to the current model. The evaluation team recommends waiting to examine program impacts until next year when the vast majority of schools implementing the current FLES model will have students in grade 3. Observations of FLES Lessons First and second grades were selected for observation because the majority of schools implement the FLES program model at these grade levels, and only a minority of schools provided FLES instruction to students above these grade levels during SY 008-09 9. This approach yielded observations in 8 of the 30 10 FLES schools with classes at these grade levels. To the extent possible, teachers were observed during instruction in a grade 1 and a grade classroom 11,1. A total of 30 teachers were observed delivering 59 FLES lessons (31 in grade 1 and 8 in grade ) in six Arabic, 13 Chinese, two French, two Japanese and 36 Spanish classrooms. In all cases, except Japanese, the observer understood the target language. The classrooms visited were randomly selected based on the observers schedules. Both FLES and regular classroom teachers were notified of the week the observations would take place, but not which classroom would be observed. The observation measure was developed by the evaluators and piloted in a Spanish classroom to assess the validity of the measure and develop inter-rater reliability between the two evaluators. Any differences in the ratings were discussed, and a consensus rating was applied. The evaluators then trained three additional staff from the Department of Accountability to conduct the observations. At the conclusion of the office training, observers accompanied the lead or co-evaluator on an observation to develop interrater reliability. The evaluators were available to answer observer questions about ratings as they arose. 7 Two schools offer FLES in kindergarten but this grade is not included in the evaluation because the program model defines FLES as beginning in grade 1. 8 Providence is phasing out Latin instruction in grades 3 to 6 during SY 008-09 and phasing in Chinese instruction in grades 1 to during SY 008-09. In keeping with the focus on implementation during the early grades, the Chinese instruction was the focus of data collection. 9 Two FLES schools, Mantua (Italian) and Daniels Run (Latin), provide FLES instruction beginning in grade 3, so were not included in data collection. 10 Two schools with FLES instruction in first and second grades were not observed because the FLES teacher at one school resigned just before the scheduled observations began and the teacher at another school was on family sick leave. 11 Several FLES teachers taught at only one grade level. One school provided FLES instruction only at Grade 1 during SY 008-09. Only first grade was observed at one school because the observers were unable to reschedule a visit to a second grade classroom before the end of the school year. 1 Brookfield and Pine Spring share a FLES teacher so only one observation was conducted at each school. J-

The FLES Classroom Observation measure was developed by the evaluators to assess the extent to which FLES teachers implement the core elements of FLES instruction described below: 1. 30 minutes of instruction;. Regular classroom teacher stays in the classroom; 3. Regular classroom teacher actively observes the lesson; 4. All students included in FLES instruction; 5. FLES instruction occurs in students regular classrooms; 6. Students respond to target language; 7. Multi-modal instructional delivery (oral, written, visual, aural, kinesthetic); and 8. Instruction in target language. Lessons were rated yes or no for each of these eight elements of FLES instruction, as well as for each individual mode of instructional delivery (item 5) 13. To be rated yes on elements, 3, 4, and 8, the element needed to be present for over 80 percent of the observation (i.e., 5 minutes of a 30-minute lesson). Observers also rated the percent of students (0%, 40%, 60%, 80%, ) who were engaged during instruction (i.e., at least 5 minutes of a 30-minute class). To gauge integration of FLES and core content Program of Studies (POS), observers indicated whether instruction addressed the FLES POS, the four core areas (language arts, mathematics, science or social studies content), and whether the core content covered was aligned with specific grade-level appropriate indicators from the FCPS POS. Lastly, the FLES observation instrument included background information, such as school, date, FLES and classroom teacher names, observer name, grade observed, and target FLES language. FLES Teacher Survey FLES Teachers at the 8 schools where observations took place were asked to respond to an online survey using Survey Monkey 14. Reminder emails were sent to ensure a high response rate. Ninety-four percent of the teachers (9 of 31 observed FLES teachers in grades 1 and/or ) responded to the survey. The survey included a total of eighteen questions, eight open-ended and ten close-ended focusing on four key areas of FLES implementation: Alignment of FLES core content; Collaboration with classroom teachers; Blackboard participation; and Professional development. FLES teachers also were asked to complete items about their own teaching efficacy, as efficacy has been found to be strongly related to student outcomes (Caprara, Vittorio, Barbaranelli, Steca & Malone, 006; Tschannen-Moran; Barr, 004) 15. In addition, teachers responded to items about their perceptions of the impact of FLES instruction on student learning, as well as suggestions for improving the program. They also completed demographic items related to their teaching experience. 13 The FLES model requires that multiple instructional modalities be employed by FLES teachers during lesson. Since oral is always used, teachers received credit for item 5 if they used oral and two additional modalities. 14 The survey was not sent to teachers at four schools; two schools with grades 1 and did not receive the survey because the teacher had resigned or the teacher was on leave (Chesterbrook and Wolftrap); two schools were not sent the survey because they do not have FLES instruction in grades 1 and (Daniels Run and Mantua). 15 Caprara, G. B., Barbaranelli, C., Steca, P. & P. S. Malone, (006). Teachers self-efficacy beliefs as determinants of job satisfaction and students academic achievement: A study at the school level. Journal of School Psychology, 44, 6, pp. 473-490. Tschannen-Moran, M. & M. Barr, (004). The relationship of collective teacher efficacy and student achievement. Leadership & Policy in schools, 3, 3, pp 189-09. J-3

Classroom Teachers at FLES Schools Emails were sent to teachers whose classes were observed requesting that they complete a survey, Survey Monkey, about FLES in their schools and classrooms. Surveys were completed for 89 percent (5 of 8) of the observed schools with grade 1 and/or. Of the 59 teachers whose classrooms were observed, 48 teachers, or 81 percent responded. The survey included both close- and open-ended questions. Mirroring the topics covered in the FLES Teacher Survey, classroom teachers provided information about alignment, collaboration and professional development. Principal Survey Principals at all of the schools implementing FLES (3 schools) were asked to complete an online survey, Survey Monkey. In June 009. Eighty-four percent of schools (7 of 3) school completed the survey. The response rate was slightly higher among schools offering instruction in grades 1 and, the focus of the current evaluation, 87 percent, and for schools that were actually observed 86 percent (4 of 8 schools). All of the respondents were principals, except in one case, where the assistant principal completed the survey. The survey included seven open-ended questions and two demographic questions. Of the 7 respondents, four did not complete the open-ended responses. The survey designed to collect information about FLES implementation in the following areas: Resource allocation; Expected FLES outcome(s); Community perspective; and Professional development. Cluster Assistant Superintendent Survey Cluster assistant superintendents responded to a question regarding the identification of FLES schools and the selection of languages. This data collection, conducted in June and July, 009, was done via email. Seven of eight clusters responded for a response rate of 88 percent. FLES Program Staff In July 009, the coordinator for World Language in the Department of Instructional Services (ISD) and the FLES resource teacher, who is charged with the day-to-day oversight of the program, participated in an interview with OPE staff. The interview addressed the following key program areas: Program model; Implementation; Resource allocation and cost optimization; Professional development; Teacher recruitment; and Program successes and challenges. Documents and Artifacts from the FLES Program and FCPS Since the beginning of SY 008-09, OPE staff have collected artifacts generated by the program, which describe, explain, organize or, in other ways, relate to the program. Most documents were provided to OPE by FLES program staff. These artifacts included: Monitoring records; PMOC FLES Progress Scope statements; PMOC FLES Project Implementation Plan; J-4

FY 008 FLES reviewed excerpts from the FCPS Program Budget; FY 009 FLES reviewed excerpts from the FCPS Program Budget; Memorandum, Peter Noonan; FLES School Appreciation memo from Peter Noonan; PEER Observation instrument; FCPS website, ISD webpage for FLES; Teacher and Principal Resources on Blackboard; and FLES Program observation notes. FLES Blackboard Lessons In July, the evaluators reviewed a sample of lessons posted to the Blackboard website by FLES teachers. The review was limited to lessons for grades 1 and. The evaluator printed the FLES Community course map from Blackboard as a complete list of lessons. In general, every tenth lesson was pulled for review. In addition, lessons were selected for teachers who posted infrequently who might not have been included in the review otherwise. At total of 50 lessons were reviewed. Each lesson plan was judged on the following specific criteria: FLES POS theme and topic identified; Identification of a language, content, and cultural objective; Posted in English; Clarity of linkages between POS topic, language objective, and content objective; Adherence to LEARN (Link, Engage/Explain, Active Learning, Reflect, Now & Then); and one integrated criterion: Overall quality of the lesson plan judged on the clarity of lesson plan, quality of content and inclusion of differentiated instruction. ISD Interview with Budget Liaison In August 009, OPE staff interviewed the ISD coordinator for Finance to obtain information on the sources of funding for FLES and the comparison program, Partial Immersion. During the interview, it was mentioned that the program manager also used her own office budget to support the programs. Consequently, a follow-up interview was conducted with the FLES program manager to document how she used her office money to supplement FLES and the Immersion activities. The information gathered to understand FLES and Immersion costs include the following: Program budget (FLES, Immersion); Grant budget (FLES only); Teacher costs (FLES, Immersion); World Languages section budget (FLES, Immersion); and Central office personnel costs. J-5

APPENDIX K FOREIGN LANGUAGE IN THE ELEMENTARY SCHOOL PROGRAM MANAGER S RESPONSE TO OFFICE OF PROGRAM EVALUATION S RECOMMENDATIONS

Foreign Language in the Elementary School Program Management Response to Office of Program Evaluation s Recommendations The Office of Program Evaluation develops recommendations for programs based on the evidence examined for the evaluation. Recommendations are intended to guide decisions about the program s future implementation. Once OPE develops recommendations, program managers are requested to provide a written response that speaks to the program s plans with regard to OPE s recommendations. The following is the program management response submitted by ISD: ISD Response to OPE s Recommendations Program Definition Ensure that the program expands its definition to include benchmarks for implementation activities, standards for student outcomes at each grade, and a system for assessing student progress between the formal assessments. o Formative assessment activities for the three modes of communication (interpretive, interpersonal and presentational) are listed for every topic in the POS published in 007. Monthly staff development sessions have focused on training teachers to assess language performance with every topic covered. o During SY 008-09, central office staff set appropriate benchmarks for each grade level beginning in grade two. Rubrics that include these expectations per grade level were rolled out to all world languages teachers at the elementary level during the fourth quarter. FLES teachers participating in the Jr. PALS field test received training in May, 009. In September, all elementary world language teachers received training on the rubric and grade-level benchmark expectations at the countywide in-service. Fidelity of Implementation Strengthen the integration of meaningful core POS content (language arts, mathematics, science, social studies) into FLES lessons. o The POS is reviewed and revised annually as the program is implemented at each grade level. There are plans to include core content objectives as used in ecart and have lesson plans more easily accessible to teachers. o ISD conducts monthly staff development sessions for all FLES teachers that are centered around content integration and the use of cooperative learning strategies that focus on language proficiency. Program Costs Monitor that funds allocated to develop a library of FLES lesson plans and resources result in an effective and usable product. o Plans to move world languages resources to ecart to allow for searchability of the lessons have been discussed, and world languages is in the cue for this move of resources. o World language office staff concur with OPE about the necessary improvement in the quality of the program resources. ISD will form a committee to approve and post lesson plans that meet the LEARN model criteria and are aligned with the POS. Each school will receive a monthly report on the lessons that FLES teachers submit to the committee. K-1

Continue to investigate alternative funding sources to defray FCPS costs for expanding FLES into additional schools. o $43,439 of grant funding included in the overall program costs for materials, supplies, equipment and staff development was supplemental to FLES program required needs and is not a cost that FCPS is expected to fund again after the grant funding ends. The Office of Language Acquisition and Title I is researching grant funding that would benefit all world languages programs K-1. Program Evaluation ISD appreciates the excellent and thorough evaluation that has been conducted by the OPE. The OPE has provided valuable feedback that will be used to strengthen and improve the FLES program. K-