Educational Development for Planning and Conducting Evaluations

Transcription

1 Year Three SE Grant ED PACE Educational Development for Planning and Conducting Evaluations

2 Goals Provide empirical data on student achievement in virtual foreign language courses using a quasiexperimental design Develop a framework for scientifically based research that can be used at the local, state, and national level to measure the impact of technology enhanced programs on student achievement

3 Partners ED The West Virginia Department of Education, drawing upon several resources within and outside of the department, created a partnership that includes: US Department of Education ROCKMAN ET AL The EdVenture Group WVEIS West Virginia Virtual School SETDA

4 Context West Virginia s Virtual School Spanish Program WV Board of Education Policy 2510 required foreign language offering in grades 7 & 8. WV Virtual School received a grant from the USDOE to develop a middle-school Spanish course, delivered via the internet. Completion of Spanish 1A (7th grade) and Spanish 1B (8th) gives students 1 year of high school credit, for Spanish I.

5 Program Design and Delivery WV Calendar daily online lesson plans interactive activities 8 Module workbooks LEAD 3 Lead Teachers Weekly phone contact Daily Lesson Plans ONLINE 15 Adjunct Teachers Grade students students CLASSROOM 32 facilitators 640 students Virtual School Internet Site WIMBA voice tools Activities and Quizzes The Virtual School Spanish program includes a 3-member instructional team made up of a lead teacher, an adjunct teacher, and a classroom facilitator. In the classroom, students engage in online activities via the VS internet site and WIMBA voice tools. students complete a variety of paper-based activities from their module notebooks. students submit work and quizzes electronically, and correspond via with their virtual amigos. once a week, students interact with the lead teacher via telephone. Outside the classroom, students complete assignments and communicate with facilitators, adjuncts, and lead teachers via .

6 Program Implementation Classes vary in size: A small class at Paw Paw A large class at Warm Springs Classes vary in location: classroom computer lab library Classes use a blended teaching model: Facilitator Adjunct Teacher Lead Teacher

7 Impact Research Summative Model Research Questions Does participation in Virtual Spanish program affect: Method Student achievement and Spanish learning? Longer term student planning / interest? Quasi-experimental Outcome Measures State standardized test Writing assessment UNIACT Interest Inventory - ACT Explore (8th) - ACT Plan (10th) Spanish assessment

8 Spanish Assessment Written Assessment Items framed on NAEP specifications for foreign language assessment Expert Panel review Background Questionnaire and Self Assessment (NAEP) Oral Assessment Adapted instrument from Center for Applied Linguistics--Student Oral Proficiency Assessment (SOPA) Training to administer SOPA (West Virginia Spanish teachers) Rater reliability review (CAL)

9 Implementation Study Research Question How is the program implemented in different sites? Method Documentation of implementation of full and sub-sample of program sites over three-year period Sample Year 1: Observed all sites in program (27) & 8 face-to-face sites Year 2: Modified / targeted study with 8 focus sites (multiple visits) Year 3: Observed all 1-B classes (26) & 5 targeted sites.

10 Implementation Research Goals: Document implementation in different settings Provide feedback about relationship between implementation and performance Provide contextual information for summative research Measures: Surveys (students, adjuncts, facilitators, all VS sites; parents, targeted sites) Observation protocol (all 26 1-B classes and 5 targeted sites) Interviews (Principal, lead teachers, adjunct teachers, facilitators, High School Spanish Teachers, and WVDE program leaders) Focus group interview (students in 1-B).

11 Research Findings From Years 2 and 3

12 Spanish Achievement Summary VS outperformed F2F On the language section of the multiple-choice assessment F2F outperformed VS On the listening and reading comprehension sections of the multiple-choice assessment On many aspects of the written Spanish assessment On all aspects of the oral Spanish assessment

13 Spanish Multiple Choice VS mean % correct s.d.(in %) F2F mean % correct s.d. (in %) VS score relative to f2f % (correct resp) Significant? (p-value of difference of means) Listening n VS = 253 n F2F = % (-0.96) Yes Language n VS = 251 n F2F = % (+0.76) Yes Reading n VS = 251 n F2F = % (-1.32) Yes 0.004

14 Spanish Writing Fluency n VS = 250 n f2f = 168 Mechanics n VS = 250 n f2f = 168 Grammar n VS = 250 n f2f = 168 Vocabulary n VS = 250 n f2f = 167 Prompt1 ideas n VS = 249 n f2f = 166 Prompt2 ideas n VS = 250 n f2f = 167 Prompt3 ideas n VS = 221 n f2f = 157 VS mean F2F mean VS score relative to f2f Significant? Yes (0.000) No (0.269) Yes (0.001) Yes (0.031) No (0.275) No (0.981) No (0.078)

15 Spanish SOPA (Oral) VS mean score (n=34) f2f mean score (n=38) VS score relative to f2f Fluency Grammar Vocabulary Listening Comprehension Significant? (p-value of difference) Yes (0.028) Yes (0.028) Yes (0.036) Yes (0.002)

16 Year 3 Study Constructs Program Design Implementation/ facilitation Intermediate Outcomes Long-Term Outcomes Roles and responsibilities of instructional team Classroom processes and activities Student engagement Spanish Learning & Proficiency Skills in reading, writing, Blended delivery model/technology and curriculum Language instruction and experiences Self-direction and responsibility speaking and listening Affective perception of learning Site administration Students use of Spanish Valuing foreign language High School Participation Performance in high Student background Collaboration and communication Student use of Spanish beyond the classroom school Spanish II Use of technology Program satisfaction

17 Preliminary Findings (student data): Students Use of Spanish: 66% of the students report using Spanish often or very often in Virtual Spanish. 50% say they listen to Spanish songs with the same frequency. 40% say they read Spanish online or in magazines. 40% write to their virtual amigos in Spanish. 33% say they speak to their classmates in Spanish. 70% speak Spanish at school (outside of Spanish class) at least sometimes. 65% speak sometimes speak Spanish at home, and at Mexican restaurants.

18 Preliminary Findings (student data): Communication and Feedback: Students communicate with their facilitator daily, and with their adjunct and telephone teachers every few days. Adjunct and telephone teachers help students correct what s wrong, answer questions, help them when they re stuck, and give them the information they need most of the time.

19 Preliminary Findings (student and facilitator survey data): Student Engagement: Students and facilitators indicate relatively high levels of engagement in Virtual Spanish activities. Over 90% of the students surveys say they do all of their assignments. 75% say they often finish their daily work before the end of the period. Over half say they often help other students with Spanish. Activities that appear to promote the highest levels of engagement include: Calendar (online) activities Speaking and listening to Spanish

20 Preliminary Findings (student data): Attitudes toward technology and online instruction: Students say they prefer online, computer-based activities to paperbased activities. In interviews, students say they would select a virtual class over a face-to-face class. Three-fourths of the students surveyed said they would like to take another virtual class. Students say computers make classes more fun and interesting.

21 Preliminary Findings (student data): Students say that the advantages of Virtual Spanish include: being able to do school work at home when they re absent being able to get ahead by working independently Over 90% of the VS 1-B students report having a home computer with internet access. Approximately two-thirds of the students say they use their home computer to access and practice VS activities. learning how to use a computer and learning a foreign language at the same time using technology to communicate with teachers and classmates

22 Preliminary Findings (student data): Attitudes toward learning a foreign language: More than 75% of the students say they learned a lot in Virtual Spanish. Approximately 90% of the students surveyed say they: like learning a foreign language think learning a foreign language is important want to continue Spanish in high school Students think speaking Spanish will prepare them for the future. It will: prepare them for more advanced classes in high school help them in college help them function in a workplace where others speak Spanish live in a more diverse world.

23 Preliminary Findings (student data): Attitudes toward learning about another culture: Students say that fiestas, cooking, holiday crafts (for, El Día de los Muertos, Cinco de Mayo, etc.) are among their favorite activities. When asked what Virtual Spanish activities they would like more of, they ask for: - more field trips to places where they can be exposed to Spanish culture - more cultural experiences - more fiestas, songs, dancing, and hands-on activities

24 Preliminary Findings (interviews with high school Spanish teachers): Students performance in high school Spanish II: Students from the Virtual Spanish program are better prepared than students who took Spanish I in high school. Language proficiency. Virtual Spanish students: have better pronunciation know more vocabulary spell Spanish words correctly have covered more material in the Virtual Spanish program learn faster and are ready to move on

25 Preliminary Findings (interviews with high school Spanish teachers): Attitudes toward class and work habits Virtual Spanish students: participate more in class feel in control take responsibility for their own learning are more conscientious value learning a foreign language

26 Preliminary Findings (interviews with high school Spanish teachers): Technology skills Virtual Spanish students: know how to find and use online language resources use technology to complete assignments have the technology skills to take other challenging classes, including AP classes which may only be available as virtual classes

27 Preliminary Findings (interviews with high school Spanish teachers): Unanticipated Outcomes Some facilitators are talking with high school Spanish teachers about the coordination between Spanish I and Spanish II/middle school and high school. Some high school Spanish teachers are adding VS activities to their own classes, and rethinking their curriculum. Some high School Spanish teachers are figuring out ways to offer additional years of Spanish or accelerated Spanish to accommodate interested students.

28 Preliminary Findings (interviews with high school Spanish teachers): Enhancing the Virtual Spanish program: Suggestions from high school Spanish teachers: - Put students impressive knowledge of vocabulary to use in writing complete sentences and putting words into context. - Help students understand the structure of sentences. - Encourage practice with stem-changing verbs and other conjugations. - Encourage reading and speaking as well as writing and listening.

29 Preliminary Findings (facilitator data): Enhancing the Virtual Spanish program Suggestions from Facilitators: Increase the Spanish class s visibility by making fiesta days, cooking, dancing, etc, a school affair. Encourage administrators to think of VS as a regular class avoid rescheduling class for other events. Encourage district support, especially in ensuring that VS students can continue Spanish II in high school without a break.

30 Preliminary Findings (student data): Enhancing the Virtual Spanish program Suggestions from Students: more group work. more opportunities to practice speaking. more computer games. more chances to see the lead and adjunct teachers how about web cams!