Acknowledgments Towards adapting the EIPA to include Cued Speech as a language option Jean C. Krause, University of South Florida Judy Kegl, University of Southern Maine Brenda Schick, University of Colorado at Boulder Consultants Judy Kegl, Brenda Schick Sam Cappiello, Hilary Franklin, Betsy Kipila Work supported in part by OSEP Grant No. H325 N010013 Programs of National Significance NCSA CST Testing Grant What is the EIPA? Specifically: Educational Interpreter Performance Assessment More broadly: A family of products focused on interpreters who work with children provided through the EIPA Diagnostic Center at Boys Town National Research Hospital (BTNRH) in Omaha, NE EIPA Products and Services The EIPA performance test EIPA Written Test and Knowledge Standards EIPA Pre-Hire Screening EIPA Code of Professional Conduct for Educational Interpreters Educational Interpreting website: www.classroominterpreting.org EIPA Performance Test Evaluates the ability to expressively interpret classroom content and discourse to receptively interpret student (child or teen) sign language Not limited to any one sign language or system; currently available to interpreters who use American Sign Language (ASL) Manually-Coded English (MCE) Pidgin Sign English (PSE) EIPA Performance Test (cont) A brief history: First EIPA materials developed in 1992 (EIPA Live ) Live classroom samples (interpreter in actual work environment) were recorded Many teams of raters in various states were trained Video Stimulus Videotapes: 2000 present Videotaped samples of classrooms and children Fewer teams of raters, all housed at BTNRH Excellent inter-rater reliability (0.93)
EIPA Performance Test (cont) Based on current theory and research (Linguistics, Child language development, Current educational practices) Widely recognized as a valid and reliable tool: Used in 30+ states Funded by the federal Office of Education as a Program of National Significance Recommended by the EIPA Review Task Force to the RID Board of Directors (June, 2006) that RID certification be granted to those who have scored 4.0 or higher and have passed the EIPA written test EIPA Format and Procedures Interpreters taking the EIPA 1. choose grade level and sign language/system: grade level: Elementary or Secondary sign language/system: ASL, PSE, MCE (current choices) 2. are provided with a warm up period for reviewing lesson plans and a video of the student 3. are videotaped during testing for later evaluation Format and Procedures (cont) Interpreting sample sent to Boys Town for analysis Boys Town evaluates within 60 days Three member evaluation team One member is deaf Complete diagnostic report Interpreters receive results of evaluation and diagnostic report $250 EIPA Test Materials Classroom tapes Option A or B Real classrooms Hearing peers are part of the classroom Cognitively challenging materials 2-5 classrooms on each testing tape Children videotapes Option A or B One child with interviewer (question prompt available) Interviewed to elicit complex language EIPA Evaluations Comprehensive rating system Interpreters rated on 40+ items, on a scale of 0 to 5 (0=no skill, 5=advanced skill) Items organized into four domains Domain scores and overall scores calculated from item scores Diagnostic feedback EIPA Rating Scale 5 excellent 4 generally good 3 generally adequate 2 not sufficient 1 very poor 0 little to no skill or no evidence of skill shown in specified category
Current project Methods Goal: to develop a Cued Speech version of EIPA that is parallel in scope, difficulty, validity, and reliability to the signing versions of the EIPA designed to evaluate the unique skills that are necessary for providing effective transliteration services using the Cued Speech system in the K-12 educational setting Interpreters taking the EIPA would choose grade level: Elementary or Secondary language/communication system: ASL, PSE, MCE, CS Designed materials for a pilot test of the Cued Speech version of EIPA completed April 2005 modified the existing EIPA video materials and scoring sheet Administered pilot test to 25 transliterators completed July 2005 Developed evaluation procedures appropriate for cueing transliterators completed July 2005 Assessed validity and reliability of pilot test completed December 2005 / June 2006 EIPA-CS Pilot Test Materials Classroom tapes (expressive, or voice-cue section) Used existing tapes No modification was required (language or communication mode of the interpreter not likely to affect the types of classroom communications that must be conveyed) Student tapes (receptive, or cue-voice section) Interviewed six deaf students whose primary mode of communication was Cued Speech Three elementary students (3 boys; ages: 10 years) Three secondary students (3 girls; ages: 13-16 years) EIPA-CS Pilot Test Materials (cont) Student tapes (cont) Interview conducted in two modalities, so that two types of cue-voice materials could be created: Audiovisual (AV), i.e. students spoke while cueing Visual-only (VO), i.e. students cued using silent mouth movements EIPA-CS Pilot Test Materials (cont) EIPA-CS Pilot Test Materials (cont) Student tapes (cont) Quantity and quality of language samples from each child were reviewed Materials deemed most appropriate were selected: one secondary girl (age: 16 yrs) one elementary boy (age: 10 yrs) Others were not usable for various reasons (student shy on camera, student speech too clear, etc.) Student tapes (cont) Warmup and test tapes were developed for the two children selected each contained an AV and a VO segment, so tapes were somewhat longer than the EIPA signing student tapes warmup tapes: ~8 minutes in length test tapes ~ 40 minutes in length
Methods Designed materials for a pilot test of the Cued Speech version of EIPA completed April 2005 modified the existing EIPA video materials and scoring sheet Administered pilot test to 25 transliterators completed July 2005 Developed evaluation procedures appropriate for cueing transliterators completed July 2005 Assessed validity and reliability of pilot test completed December 2005 / June 2006 EIPA-CS Test Administration 25 transliterators from four states two states participated formally and had 21 transliterators tested between them four individuals from two states opted to be tested EIPA-CS Test Administration (cont) 14 transliterators chose elementary level 7 selected Option A classrooms 7 selected Option B classrooms all selected Option A student cuer (only option) 11 transliterators chose secondary level 7 selected Option A classrooms 4 selected Option B classrooms all selected Option A student cuer (only option) The Test Selection of materials and test format Elementary or Secondary Classroom A or Classroom B (for expressive) Cuer A (for receptive) no choice in pilot Forty-five minute warmup period Booklet describing classrooms in more detail Video of Cuer A to practice cue-voice The Test (cont) Methods Receptive portion (~40 minutes) ~20 minutes audiovisual ~20 minutes visual-only Expressive portion (~40 minutes) Elementary: 5 classrooms, about 7-8 minutes each Secondary: 2 classrooms, about 20 minutes each Designed materials for a pilot test of the Cued Speech version of EIPA completed April 2005 modified the existing EIPA video materials and scoring sheet Administered pilot test to 25 transliterators completed July 2005 Developed evaluation procedures appropriate for cueing transliterators completed July 2005 Assessed validity and reliability of pilot test completed December 2005 / June 2006
EIPA-CS Evaluation Procedures Modification of score sheet (cont) Two steps: Modification of the existing EIPA score sheet Development of reliable evaluation procedures for each item on the score sheet For both steps: Initial draft was created by EIPA-CS development team Additional input from three expert consultants to ensure that the final skill list was comprehensive 2 transliterators, 1 consumer all three were certified Instructors of Cued Speech all three were pursuing, had completed, or planned to complete Masters degrees Some items were appropriate and remained Example: Stress/emphasis for important words or phrases (Roman I) Other items were inappropriate and removed Example: Location/relationship using ASL classifier system (Roman I) A number of CS-related items were introduced Example: Appropriate use of alternate cueing hands (Roman I) Modification of score sheet Some changes to domains were required as well Roman I Roman II Roman III Roman IV Interpreter product voice-to-sign Interpreter product sign-to-voice Vocabulary Overall Factors EIPA (sign versions) EIPA-CS Pilot Test Interpreter product voice-to-cue Interpreter product cue-to-voice (AV and VO rated separately) Intelligibility Overall Factors Roman I: Voice-to-Cue Prosodic information: A. Stress/emphasis for important words or phrases B. Affect/emotions (appropriately uses face and body) C. Register (interpreter matches speaker s register) D. Sentence/clausal boundaries (no run-ons) E. Sentence types indicated Other supporting information: F. Use of space, natural gestures, eye gaze, and body shifts G. Identification of speaker and other sound sources H. Communication of meaningful environmental sounds I. Appropriate use of alternate cueing hands Interpreter performance: J. Awareness and self-correction of cueing errors Roman II: Interpreter Product Cue-to-Voice (AV & VO) Can read and convey student s A. Cued words B. Proper names, unusual vocabulary C. Register (if applicable) Vocal / Intonational features: D. Speech production E. Sentence/clausal boundaries F. Sentence types G. Emphasize important words, phrases, affect/emotions Interpreter performance: H. Adds no extraneous words/sounds to message Roman III: Intelligibility Cue accuracy: A. Appropriate selection of cues (handshapes and placements) B. Representation of dialects/alternate pronunciations C. No extraneous cues Clarity of cues: D. Appropriate formation of handshapes E. Appropriate locations for placements F. Appropriate execution of specified movements G. No extraneous movements or distracting physical features Clarity of oral information: H. Appropriate visibility of articulators I. No extraneous oral mannerisms or distracting facial features Timing: J. Fluency (rhythm and rate; use of liaisons) K. Synchronization between cues and mouth movements
Roman IV: Overall Factors Message processing Voice-to-Cue (V-C): A. Preserves a sense of the whole message (V-C) B. Keeps pace with speaker (V-C) C. Uses verbatim transliteration and paraphrases appropriately (V-C) Message processing Cue-to-Voice (C-V): D. Preserves a sense of the whole message (C-V) E. Demonstrates process decalage (lag time) appropriately (C-V) F. Uses verbatim transliteration and paraphrases appropriately (C-V) Evaluation Procedures Expert CS consultants finalized evaluation rubrics for each item and served as an evaluation team An additional evaluation team was trained Most pilot tests rated multiple times to assess reliability intra-rater reliability (consistency of one team s ratings over time) inter-rater reliability (consistency of ratings between two different teams) Rating timeline Team 1 Initial meeting: rated 12 test tapes (6 elementary, 6 secondary) After 3 weeks: rated 14 test tapes (7 elementary, 7 secondary) 6 tapes had been rated previously (3 elementary, 3 secondary) Repeats allow for intra-rater (i.e. intra-team) reliability, early in the development process Rating timeline (cont) Team 1 (cont) After 3 more weeks: rated 7 tapes (4 elementary, 3 secondary) all had been previously rated at the second meeting Repeats allow for intra-rater (i.e. intra-team) reliability, later in the development process In total, 33 scores assigned to 20 test tapes (13 repeats within Team 1) Rating timeline (cont) Team 2 rated 12 test tapes (7 elementary, 5 secondary) 7 tapes had been rated previously (3 elementary, 4 secondary) by Team 1 Repeats allow for inter-rater (i.e. inter-team) reliability In total, 45 scores assigned to 25 test tapes (13 repeats within teams, 7 repeats across teams) Methods Designed materials for a pilot test of the Cued Speech version of EIPA completed April 2005 modified the existing EIPA video materials and scoring sheet Administered pilot test to 25 transliterators completed July 2005 Developed evaluation procedures appropriate for cueing transliterators completed July 2005 Assessed validity and reliability of pilot test completed December 2005 / June 2006
Results Results (cont) EIPA-CS Rating 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 1 3 5 7 9 11 13 15 17 19 21 23 25 Transliterator Average scores for the 25 pilot tests ranged from 2.4 to 4.6 However, the 45 actual scores assigned by the two teams ranged from 2.1 to 4.6 Mean score was 3.39 (σ = 0.58) Results (cont) Domain averages for 45 scores assigned: Roman I (Interpreter product voice-to-cue): 3.47 (σ = 0.77) Roman II-AV (Interpreter product Audiovisual cue-to-voice): 3.33 (σ = 1.02) Roman II-VO (Interpreter product Visual-only cue-to-voice): 1.69 (σ = 1.38) Roman III (Intelligibility): 4.16 (σ = 0.44) Roman IV (Overall Factors): 3.00 (σ = 0.88) Results (cont) Average scores were roughly equivalent across level and Classroom option Elementary level (24 scores, 14 transliterators): 3.31 (σ = 0.47) Classroom A option (12 scores, 7 transliterators): 3.41 (σ = 0.56) Classroom B option (12 scores, 7 transliterators): 3.21 (σ = 0.36) Secondary level (21 scores, 11 transliterators): 3.55 (σ = 0.69) Classroom A option (12 scores, 7 transliterators): 3.45 (σ = 0.71) Classroom B option (9 scores, 4 transliterators): 3.68 (σ = 0.68) Test is equivalent in difficulty across level and Classroom Option for Cued Speech, just as it is for the sign language options Validity Assessment Face validity - good Determined by judgment of expert consultants Overall scores and domain scores were reviewed at the end of each rating session Numerical scores matched qualitative assessments of transliterators Content validity good Determined by judgment of expert consultants Test is comprehensive in covering skills needed for successful performance in the classroom Validity Assessment (cont) Criterion validity more data needed Compare to known instruments (e.g. state tests, national certifications such as CLTNCE offered through TECUnit) Early results (2 subjects) are promising; more data needed This will be analyzed over time
Reliability assessment Preliminary results presented today do not reflect most current data analysis likely to improve due to final modifications in scoring completed in June 2006 Overall reliability calculated via two methods: Chi-square tests for the goodness of fit: evaluated the hypothesis that the two sets of scores obtained for a given set of tests had equal means and variances (strictly parallel model) Interclass correlation coefficients: measured the amount of agreement between pairs of scores Reliability assessment (cont) Chi-square tests no statistically significant evidence to suggest that tests were scored differently by the two different teams (p=0.324) or by Team 1 over time, either early (p=0.843) or late (p=0.388) same score no matter when the test is rated or which team rates it Reliability assessment (cont) Interclass correlation coefficients showed High intra-rater reliability, both early and late Early: ICC=0.933, p=0.006 (statistically significant) Late: ICC=0.816, p=0.030 (statistically significant) Moderate inter-rater reliability (ICC=0.615) Effect was not statistically significant (p=0.104) Trend is evident, but more inter-rater reliability data needed in order to verify whether this correlation is an accurate reflection of the inter-rater reliability Reliability assessment (cont) Reliability of scores for each of the five area scores (Roman I, Roman II(AV), Roman II(VO), Roman III, and Roman IV) was assessed in the same manner similar results Conclusions EIPA-CS pilot test successfully developed, administered and evaluated high face and content validity high intra-rater reliability moderate inter-rater reliability is likely, but not yet confirmed Conclusions Data collected in pilot work may useful for research purposes areas in need of professional development can be readily identified, e.g.: Strong need for visual-only cue-to-voice skill development Moderate need for skill development in providing prosodic and other supporting information in order to relay a complete sense of the message
Status Talks are underway with BTNRH to determine whether and when the EIPA Diagnostic Center can begin offering a CS version of the test alongside ASL, MCE, and PSE Pilot test data is good, so there is reason to be optimistic No definite plans to do this yet, but decision can be expected sometime this fall Next steps Currently BTNRH is working on updating their database to include the CS evaluation items This will allow results for EIPA-CS pilot tests to be officially registered at BTNRH Results are considered official results on the Cued Speech pilot test Participants will receive an official notice from BTNRH to this effect when database work is complete Next steps (cont) In the medium term, Practice materials for voice-to-cue will be developed for the EIPA website Additional validity and reliability data will be collected Additional environmental sounds may be added to test tapes Next steps (cont) In the longer term (if additional funding is obtained), EIPA: Written test could be adapted for Cued Speech Option B elementary and secondary students could be developed for cue-tovoice portion of the test