Summary of the Case University of Virginia Teacher Education Program 1

Transcription

1 Summary of the Case University of Virginia Teacher Education Program 1 The Summary of the Case is written by the auditors and approved by program faculty. The Summary reflects the auditors understanding of the case the faculty are making for accreditation. Authorship and approval of the Inquiry Brief: The Inquiry Brief was written by Dr. Linda Boone in collaboration with the teacher education coordinators and faculty in Elementary Education, Secondary Education, and Special Education within the Curriculum and Instruction & Special Education Department. The Inquiry Brief was approved by the Coordinators of the three divisions and department chair as representatives of the faculty on June 11, Introduction: Founded in 1819 by Thomas Jefferson, the University of Virginia (U.Va.) is a highly selective public, non-profit institution of higher education located in Charlottesville, Virginia. The University is dedicated to Jefferson s democratic belief in developing leadership through education. A student population of approximately 24,000 individuals undertakes studies at the undergraduate, masters, and doctoral levels in dozens of fields, not only in Charlottesville but also through partnership programs throughout the US and foreign countries. Admission is selective and competitive. According to the U.Va. website, the University has never been lower than second among public universities ranked by U.S. News rankings, and it has never dropped out of the top 25 listing of all national universities. The Teacher Education Program (TEd@Curry) is located in the University of Virginia s Curry School of Education. The program focuses on the Teacher as a Reflective Decision-Maker, integrating field work with professional coursework to prepare candidates to connect content, practice and research into best practices that they can use to inform instructional decision-making. The program operates within the department of Curriculum, Instruction, and Special Education (CISE), incorporating faculty specializing in Elementary Education (ELED), Secondary Education (SEC), and Special Education (SPED). TEd@Curry offers two related degree options: a five-year program in which students earn a Bachelors Degree in an academic subject as well as a Master of Teaching degree (B/MT) and a related post-graduate option in which students enter following completion of a Bachelors degree and then complete the remaining requirements for the Master of Teaching (PG/MT). Satisfactory completion of either option results in recommendation for the post-graduate professional teaching license from the Virginia Department of Education (VDOE). Field experiences and student teaching in the options are generally undertaken in the Charlottesville area, either in Charlottesville Public Schools or in the surrounding Albemarle County Public Schools, although it is possible for students to take advantage of partnerships with other school systems in Virginia or other states, as well as international study in the UK in Belfast, Northern Ireland and in Cambridge, England.

2 The Teacher Education Program consists of 32 full-time and 6 part-time faculty members. The program graduated 136 students in 2013 (91 B/MT and 45 PG/MT) in 2013 and enrolled a total of 362 students in in the following options: Table 1 Numbers of Students in the University of Virginia Teacher Education Program Options, Option Name Number of B/MT completers in previous academic year (2013) Number of PG/MT completers in previous academic year (2013) Number of B/MT students enrolled in current academic year ( ) Number of PG/MT students enrolled in current academic year ( ) Elementary Education (ELED) (PK-6) English Education (6-12) Mathematics Education (6-12) Biology Education (6-12) Chemistry Education (6-12) Earth Science Education (6-12) Physics Education (6-12) Social Studies Education (6-12) French Education (PK-12) German Education (PK-12) Latin Education (PK-12) Spanish Education (PK-12) Health & Physical Education (PK-12) Early Childhood Special Education (ECSE) General Curriculum (SPED/ELED) General Curriculum (SPED/SEC) General Curriculum (SPED/ELL) Add-on Endorsements 3 & 4 Year Olds (add-on) Algebra I (add-on) English as a Second Language (ESOL) (add-on) Gifted Education (add-on) Program claims: The Teacher Education Program faculty makes three broad claims about its graduates, with subsets of the claims regarding the graduates performance: Claim A: Curry teacher education (TEd@Curry) graduates are qualified to teach under the Commonwealth of Virginia s regulations for teacher licensure. TEd@Curry graduates: a. demonstrate knowledge and competency in liberal arts and sciences that are foundational for any teaching professional. b. demonstrate subject competency in their chosen endorsement area(s) of

3 teacher licensure. c. have successfully applied evidence based pedagogical methods of instruction in multiple educational environments. d. contribute to prek-12 student achievement through appropriate use of assessments and instructional decision making. e. are proficient in the application of educational technology for instruction. Claim B: Curry teacher education graduates have the dispositions, knowledge, and skills to work with students with diverse backgrounds and diverse needs. graduates: a. employ student engagement techniques that respect individual differences. b. work effectively with others to support student learning. c. show sensitivity in building student-teacher relationships and creating positive classroom environments. Claim C: Curry teacher education graduates utilize data and reflection to improve their understanding and application of evidence- based practices. Evidence supporting the claims The faculty identifies nine types of evidence in support of its claims. As a faculty particularly concerned with contributions to research on valid and reliable measures of teaching, the TEd@Curry faculty chose only to use measures with acceptable levels of reliability. The individual measures include: 1. Graduate Record Examination (GRE) scores (Claim A - content knowledge) GRE scores are used for entrance into the program. They are used in the Inquiry Brief as measures of liberal arts and science knowledge and also subject-matter knowledge. GRE scores for all three subtests (verbal reasoning, quantitative reasoning, and writing) were analyzed separately for B/MT and PG/MT entrants into the classes of , relying on the 2013 publication of the Educational Testing Service (ETS) for the subtests validity and reliability. Candidates are expected to reach qualifying scores (V-150, Q-145, W-4.0) for Program entrance, although the faculty can waive the cut score requirement if a candidate s overall record shows strength. Results for 636 TEd@Curry students accepted into the classes of show overall means and standard deviations of V= (5.50), Q= (5.77), and W=4.47 (0.64). 2. Virginia Communication and Literacy Assessment scores (Claim A - content knowledge) The Virginia Department of Education (VDOE) requires all candidates for initial certification to pass the Virginia Communication and Literacy Assessment (VCLA). The test was developed and validated by the Evaluation Systems Group of Pearson, Inc. and adopted by the Commonwealth of Virginia in Reliability is reported on the Pearson website. The scaled passing scores are 235 for the reading subtest, a score of 235 for the writing subtest (or a composite score of 470 for the both subtests). Most students complete the tests after admission. Candidate scores are

4 stored in U.Va. s Student Information System (SIS); only the highest score for each candidate was used in the analysis. VCLA results for 612 students in the classes of are reported in the composite and as subsets by program option (ELED students, SEC students, and SPED - 88 students). Overall, the VCLA means and standard deviations were (8.68) for the reading subtest, and (10.82) for the writing subtest. 3. Praxis II scores (Claim A - content knowledge) The Praxis II examination is also required by VDOE for initial certification, and candidates must also pass the test for TEd@Curry program completion. The test was developed and validated by ETS. The faculty analyzed test scores for all candidates from Cut-scores established by VDOE vary depending on the certification area, so the faculty determined not only the means and standard deviations, but also the effect size (i.e., minimum deviation from the Virginia mean) for candidates from each certificate area, settling on a published effect size of 0.80 standard deviation above the state mean as a standard for the mean scores. Praxis II mean scores for 535 students from varied from (7.81) for 6-12 English candidates to (0.74) for 6-12 Physics (2 students). All means were well above the VDOE cut scores, and all but one (Physics) had an effect size greater than the faculty s selected effect size of 0.80 standard deviations above the state mean. 4. Classroom Assessment Scoring System - CLASS (Claims A, B, C) Rather than relying on its student teaching rubric (about which the faculty has continuing concerns regarding its reliability), the faculty has chosen to rely on its Classroom Assessment Scoring System (CLASS), a teaching analysis instrument with solid validity and reliability developed by researchers at U.Va. over the past decade. CLASS is composed of 10 high-inference scales aligned within three domains of teaching: emotional support, classroom organization, and instructional support. Its reliability was reported in 2009 (Pianta and Hamre, 2009). Trained supervisors observe two separate videos of student teacher classroom performance and code their agreement with item-statements about the student teacher s performance on a 7-point Likert-type scale (1-2=low, 3-5=middle, and 6-7=high). The faculty s expectation is that the mean scores for each domain should be greater than 3.5. Because not all supervisors had evidenced reliability in coding the scales, only 59% of the student teachers (called teaching associates at TEd@Curry) from were included in the analysis. Mean ratings for the classroom organization and instructional support domains were used to assess pedagogical knowledge, while the mean ratings for emotional support were used to assess caring and effective teaching skill. Mean ratings and standard deviations for the 368 student teachers (teaching associates) were 5.00 (0.78) for classroom organization and 3.89 (0.99) for instructional support. The mean rating and standard deviation for the same 368 teaching associates for emotional support was 5.39 (0.62). 5. Grades and GPA (Claim A - content knowledge; teaching with technology)

5 The faculty chose to examine grades in two ways: the mean GPA in the academic major as a means of confirming students content knowledge, and the mean grade in EDIS 3450 (Teaching with Technology), a required course for all program students. The faculty reasons that the validity of grades has been established through consensus development over years. At U.Va., grades are assigned using letters (A, B, C, etc.), and they are translated into a grading system. The TEd@Curry program faculty has set a grade of B- (2.7) as the requirement for credit for all courses and an undergraduate GPA of 3.0 as a cut-score for program entrance. With regard to the GPA in the major, the faculty uses only the B/MT undergraduate major GPA at the semester prior to graduation at the Bachelors level in their analysis; however, due to the way in which the University s Student Information System operates, the faculty was able to access only B/MT student major GPAs for the Class of The undergraduate major GPA for PG/MT students was not analyzed because it is not always provided on entering student transcripts. Mean major GPAs and standard deviations for 88 B/MT students in the Class of 2013 were 3.47 (0.30) for SEC, 3.39 (0.42) for ELED, and 3.38 (0.47) for SPED, all above the faculty s cut-score of 3.0. However, the faculty expressed concern regarding the percentages of candidates meeting the expectation. While 95% of SEC candidates had a Major GPA of 3.0 or higher, only 87% of ELED candidates met the requirement and 75% of SPED candidates met requirements (86% of the class as a whole met requirements). The faculty found that candidates with low major GPAs were admitted primarily in recognition of either one particularly bad semester or a steady improvement over time in course grades. With regard to the grades in EDIS 3450 (Teaching with Technology), this is a project based course aligned to the importance of teachers knowledge for technology integration or technology, pedagogy, and content knowledge (TPACK) and one consistent across all program areas. The faculty expectation is a mean grade of 3.0 for all students in the course, Of 458 student grades analyzed, means were 3.89 (SEC), 3.97 (ELED), and 3.90 (SPED). The faculty reported that with the exception of a few outliers each year, graduates consistently meet expectations in this course. Only one candidate in the past five years has failed the course, and that person met the requirement by transferring in an accepted credit from another university. 6. Lesson Plan Rubric (Claim A - teaching with technology) A program-wide lesson plan rubric was piloted in Following minor revision, it was re-piloted in one pre-student teaching course in Spring 2013 to cover six planning areas including lesson goals and aligned objectives, assessment, instructional procedures, preparation for the lesson, responsiveness to student needs, and use of technology. The rubric uses a rating scale of 4=Exemplary; 3=Proficient; 2=Needs improvement; 1=Unacceptable. For Use of technology on the rubric, 0=Not Applicable was added. For the Inquiry Brief self-study, expectations were that no candidate would need improvement or provide unacceptable planning in the category of Use of Technology. For construct validity,

6 the rubric was developed making use of published commonalities of lesson plans. Currently, the rubric is completed by only one rater at a time; training of additional rater and reliability studies have yet to be undertaken. In the first application of the rubric in the Spring 2013 pilot (N=7), 86% were rated as proficient and the remaining 14% were rated as needs improvement. In the second application in Spring 2013 (N=13), 15% were rated as exemplary and 69% were rated as proficient (while the remaining 15% were rated as not applicable ). Thus no candidates were rated at the unacceptable level. 7. Student Survey Analysis Rubric (Claim C - using data and reflection) TEd@Curry student teachers are required to survey their K-12 students, analyze the data from the survey, and write a reflective paper on the results. The survey form varies according to grade level, but all versions are modeled on samples provided by the VDOE and also approved by the faculty. For the Class of 2013, the resulting paper was graded using a rubric by the teaching associate seminar instructor, and then 10% of the papers were re-coded by an external rater on a rubric that covers reflection on knowledge of subject, reflection on pedagogical knowledge, caring and responsive teacher, and improvement steps. The scale used is similar to that of other tools: 4=Exemplary, 3=Proficient, 2=Needs Improvement, 1=Unacceptable; the faculty expectation is that candidates will score 3 or higher on all analysis sections of the assessment. Reliability for the assessment in was analyzed using Cohen s Kappa and inter-rater reliability. The kappas ranged between.29 k.62, indicating a fair agreement or a substantial agreement by chance between raters. Inter-rater agreement was 100%, indicating strong agreement. For the 25 student teachers in the 2013 pilot, means and standard deviations were 3.00 (0.80) for Knowledge of Subject, 3.12 (0.77) for Pedagogical Knowledge, 3.24 (0.76) for Caring and Responsive Teacher, and 3.20 (0.94) for Improvement Steps. Although the small sample size restricted the faculty from drawing valid conclusions, the faculty were encouraged by the strength shown in candidates ability to articulate improvement steps based on their review of the data from their students, indicating an ability to learn from the data gathered. 8. Field Project Paper Rubric (Claim C - using data and reflection) A Field Project (mastery-based project) is completed by each MT student during the final semester of the program, after student teaching has been concluded. The project requires candidates to identify three K-12 students to observe and to conduct interviews with their teachers based on a case study model suggested in the literature. Each student s project report is graded using a locally-developed analytic rubric based on a four-point scale (4-Exemplary, 3-Good, 2-Needs Improvement, 1- Unacceptable), and data from four separate categories of the rubric were investigated: data collection, findings, discussion, and recommendations. The expectation for this assessment is that candidates will score an average of 3.5 on these four categories. Multiple raters graded each project report, and the raters had achieved consistency of rating through a series of sessions grading the reports and discussing their ratings. Due to a recent change in data collection to irubric, only the results from the were analyzed. Means and standard deviations for

7 the four categories of the project reports completed by 93 MT students in varied between ( ). Thus the results met the faculty s expectation for all four categories, although the faculty recognize concern that improvements can be made in the future regarding the students use of data and their reflections on its implications. 9. Disposition Scales (Claim A - pedagogical knowledge & teaching skill, and Claim B - pedagogical knowledge, teaching skill, and multicultural perspectives and accuracy, and Claim C pedagogical knowledge) The TEd@Curry faculty employed four different instruments to measure student dispositions throughout their time in the program: The Teacher Self-Efficacy Scale (TSE), the Ideas about Children Scale (IAC), the Teacher Multicultural Awareness Scale (TMAS), and the Culturally Responsive Teacher Self-Efficacy Scale (CRTSE). These scales are incorporated into TEd@Curry s Common Core of scales and surveys completed by the Curry Participant Pool (CPP) of current students as a part of Curry s on-going faculty research on teacher dispositions. Each student is required to participate in a number of surveys each year, and most (but not all) complete the four scales listed above. Participant data from all five years ( ) were assessed and analyzed for each of the four scales, as explained below: The Teacher Self-Efficacy Scale (TSE) is a 24-item survey investigating the respondents perceived efficacy in three categories: instructional strategies, classroom management, and student engagement. Participants respond on a 9- point Likert-type scale (1-nothing, 3-very little, 5-some influence, 7-quite a bit, and 9- a great deal). Validity and reliability information for the TSE is referenced, but not included in the Inquiry Brief. The overall mean and standard deviation for 494 students ( ) for the TSE scale for instructional strategies were 7.37 (0.95), with very little variation among the different cohorts (SEC, ELED, and SPED) for any of the five years. While the faculty did not state a specific numerical expectation for the TSE means (they expect candidates to identify on the positive end of the scale), they stated that they were satisfied with the outcomes. The Ideas about Children Scale (IAC), also called the Modernity Scale, was first used to examine parental views about children, investigating the extent to which views are child-centered rather than adult-centered (authoritarian). The scale was later redeveloped to investigate the views of teachers about children and more recently the views of student teachers about children. Participants respond to the 16-item survey on a 5-point Likert-type scale (5-strongly agree to 1-strongly disagree). Validity and reliability information for the IAC is referenced, but not included in the Inquiry Brief. The overall mean and standard deviation for 496 students ( ) for the IAC Scale were 2.17 (0.45), with all individual means for SEC, ELED, and SPED between The faculty stated that its expectation for the IAC means (as with all disposition scales) would show candidates identify on the positive end of the scale. They stated that the outcome was positive and that the consistency of the means provided a measure of reliability.

8 The Teacher Multicultural Awareness Scale (TMAS) is a 20-question survey uses a Likert-type scale (5-strongly agree to 1-strongly disagree) with higher scores indicating increased multicultural awareness and sensitivity. The results from the 20 questions are averaged to create a composite score. The faculty set a composite mean composite score of 3.4 as its expectation. Validity and reliability information for the TMAS is referenced, but not included in the Inquiry Brief. The overall mean and standard deviation for 489 students ( ) for the TMAS Scale were 3.95 (0.42), with all individual means for SEC, ELED, and SPED between Based on this, the faculty stated that no cohort has failed to meet expectation regarding multicultural awareness throughout the five-year period, while noting some recent weaker numbers. The Culturally Responsive Teacher Self-Efficacy Scale (CRTSE), first published in 2006, examines participants culturally responsive teaching self-efficacy and outcome expectancy beliefs. Participants rate their confidence in being able to accomplish 41 different teaching tasks related to their cultural self-efficacy on a scale with 3 defined points (100-completely confident, 50-moderately confident, 0-no confidence at all). The 41 ratings are averaged to create a composite score. Validity and reliability information for the CRTSE is referenced, but not included in the Inquiry Brief. The overall mean and standard deviation for 488 students ( ) for the CRTSE scale were (12.2), with all individual means for SEC, ELED, and SPED between , consistent with the expectation that candidates would identify on the positive end of the scale. The faculty noted that ELED students consistently score higher on the CRTSE than SEC students. Based on the results of the 9 types of assessments, the faculty stated that its graduates meet their claims as expressed publicly. Internal audit: The internal audit was organized according to a schematic of the TEd@Curry Quality Control System developed to account for TEAC components and subcomponents in Principle 3 (Institutional Capacity). (See Fig. 4 in the Inquiry Brief on page 38.) Because several aspects of the data management systems at U.Va. have been revised and updated recently, the faculty chose to select a randomized sample of 10% of the current candidates in each program option to use as the starting points for the internal audit probes: Special Education (4 candidates), Elementary Education (8 candidates), and Secondary Education (10 candidates). In addition, 10% of recent graduates (13 alumni) were randomly selected for probes related to graduates. TEd@Curry graduate students and staff members completed the audit using available U.Va. and TEd@Curry databases, websites, publications, and records. Findings were reviewed by the four TEd@Curry coordinators. The internal audit team investigated 16 probes and found the Quality Control System to be working as essentially designed. In its discussion of the findings, the faculty noted the usefulness of the process. The faculty also expressed a need to address mechanisms to assure the quality of on-line learning, a need to address the documentation and procedures for tenure/promotion, and more attention to the transition of candidates from pre-service to in-service professionals.

9 Plans for program improvement While generally satisfied with the outcomes of their self-study, the faculty described the need for program improvement in the following areas: Continuing development of statistical reliability and validity of program assessments Investigating graduates performance in a more systematic way, in particular adapting and validating research techniques regarding the impact of program completers on K-12 student learning Obtaining accurate feedback from alumni, not only as a means for TEd@Curry program improvement in the future but also as a means of developing more valid and reliable alternatives to state-mandated mechanisms to link K-12 teachers to their preparation programs. Investigating and developing the usefulness and reliability of My Teaching Partner (MTP), a new form of video-based assessment for student teachers being developed at U.Va. Examining differences among B/MT and PG/MT outcomes Revisiting the evaluation tools used at different levels of the program Streamlining the data collection process by contracting with an external online assessment system Statement regarding commitment and capacity: The faculty concluded that The University of Virginia is committed to Teacher Education Program and that there is sufficient capacity to offer a quality program. The faculty highlighted its interest (and need) to attend to the new CAEP standards as it continues its ongoing self-study and program improvement. 1 The University of Virginia Teacher Education Program offers options at the undergraduate/masters and post-graduate/masters level in PK-6 Elementary Education, 6-12 Secondary Education (English, Mathematics, Biology, Chemistry, Earth Science, Physics, and Social Studies), PK-12 Foreign Language (French, German, Latin, and Spanish), PK-12 Health & Physical Education, Special Education (Early Childhood and General Curriculum Dual Endorsement in Elementary Education, Secondary Education, and English Language Learners), and add-on endorsements for 3 & 4 Year Olds, Algebra I, English as a Second Language (ESOL), and Gifted Education. The state of Virginia, at its discretion, offers licensure to program completers in these option areas. Acceptance of the Summary of the Case The faculty accepted the Summary of the Case as accurate on October 21, 2014.