FACTOR Anlysis of Intern Effectiveness Sid T. Womck, Ph.D. Shellie Louise Hnnh, Ed.D. Columbus Dvid Bell, Ed. D. Arknss Tech University Four fctors in teching intern effectiveness, s mesured by Prxis III-similr instrument, were found mong observtionl dt of teching interns during the 2010 spring semester. Those fctors were lesson plnning, techer/student reflection, firness & sfe environment, nd professionlism/efficcy. This fctor nlysis ws s much of sttement bout effective teching s it is bout the technicl spects of n instrument utilized to ssess it. Forty-one percent of effective teching ws found to be in the lesson plnning. Keywords: effective teching, supervision of interns, efficcy, sfe school environment, techer reflection, higher order thinking, NCATE Stndrd One, novice techers, observtion systems At our university, we re constntly looking for wys to help techer eduction cndidtes improve their teching. As is probbly the cse in most techer eduction units in the United Sttes, our College of Eduction uses n observtion form for ssessing techer intern performnce nd for giving feedbck. When the Formtive Observtion nd Intervention form ws creted severl yers go, it ws constructed so tht items nd domins hd gret resemblnce to the Pthwise evlution (ETS, 1996). Accordingly, out of respect for intellectul property rights, we obtined written permission from the Eductionl Testing Service before beginning to use it with our cndidtes. This form hs become useful not only for ssessing intern performnce, but lso for identifying the most slient elements of effective teching. In other words, the form identifies wht is relly being identified s effective in teching. Pthwise ws developed through Eductionl Testing Service s n observtion system to gther rich, reserchbsed, objective clssroom dt bsed on evidence stemming from the effective teching reserch (Chn, 1998). The effectiveness of techers during clssroom settings is rted s ctegory one, ctegory two, or ctegory three, depending upon very specific scoring criteri (ETS, 1996), with ctegory one denoting n uncceptble level of effectiveness. The ssessment of teching competency is thus very uthentic portryl of teching performnce, since minimum of subjectivity is employed. In ddition to the 19 hevily reserch-bsed items relted to the Pthwise system, two items were dded loclly for dministrtive nd prgmtic resons: one under Domin A, to denote totl prepredness to tech, nd nother under Domin D, to denote the cndidte s consistency in meeting professionl responsibilities. The observtion form ws used to collect dt on 21 reserch-bsed items of techer performnce. These 21 res were grouped into four domins: (A) Orgnizing Content for Student Lerning; (B) Creting n Environment for Student Lerning; (C) Teching for Student Lerning; nd (D) Techer Professionlism. The items (not yet the fctors) of the observtion form re shown in Tble 1. Since the dt obtined using the Formtive Observtion nd Intervention form were used to mke personnel decisions bout cndidtes, we decided to study it in depth, using cndidte dt from the Spring Semester of 2010. We felt tht, by doing this study, we could gin insight into the chrcteristics of effective teching in ddition to exploring some technicl spects of the instrument. A priori ssumptions Fctor nlysis cn be used to test whether initil ssumptions bout fctor structure of n ssessment instrument hve empiricl vlidity. Our ssumptions were s follows: 1. Four fctors would be found, corresponding to the four domins of Pthwise.
VOLUME 2, ISSUE 1 2. The items tht mesured these fctors would be locted within the domin structure suggested by Pthwise. 3. The two items tht hd been dded loclly would not lod (correlte) significntly upon the rest of the fctor structure. 4. Decisions bout the fctor structure would not be bsed hevily upon the two loclly-developed items lone. Definitions Domin: A collection of five or more items on the Formtive Observtion nd Intervention form designed to ssess the sme construct. The number of items on the form exceeded the minimum number of three items to crete component (expected fctor), s described by Htcher nd Stepnski (p. 460). Effective teching: An ssessment of teching using the Formtive Observtion nd Intervention form (sometimes referred to simply s the form) which yielded mesurements of 2 or bove in every one of 21 items on the form. Teching ws not regrded s effective if there ws not enough evidence during n observtion to support ctegory of t lest 2 in ech nd every one of the 21 items. Fctor: A mthemticl communlity with n Eigenvlue of t lest 1. On the Formtive Observtion nd Intervention form, mthemticl communlity tht ccounted for t lest 1/21st of the vrince of the entire 21-item instrument used to mesure techer effectiveness. Fctor nme: The nme given to collection of items from the form identified during the fctor nlysis process whose items hve sttisticlly significnt (p<.01, n=130, one-tiled test) (Ferguson, p. 494) correltion to the fctor nd which seem to best typify the construct of the five items most correlted to the fctor. Purpose of the Study The principl purpose of our study ws to determine if there were fctor lodings on this mesure of effective teching nd, if principl fctors were found, to determine wht those fctors were by crefully ssigning nmes to them. Method Htchett nd Stepnski (1994, p. 461) stte tht for fctor nlysis, the smple size should be the lrger of 100 subjects or five times the number of vribles being nlyzed. Five times 21 items is 105. There were 130 teching interns in the smple, nd 416 teching observtions recorded, so the smple ws more thn dequte in size to ccommodte this type of nlysis. Methodologiclly this study should be considered common fctor nlysis (Ingrm, 2011). Prticipnts Prticipnts were 63 erly childhood, 9 middle level, nd 58 secondry eduction interns, totl of 130 senior intern cndidtes. They were ssigned to school cmpuses in the Western prt of Arknss, prticulrly long the I-40 corridor from Morrilton westwrd to the Arknss-Oklhom stte line. All were ssigned to ccredited public schools nd in content res pproprite to their mjors nd expected licensures. Plcement ws done through the office of Techer Eduction Student Services t the university. All public school nd university fculty who prticipted in ny direct wy in intern evlutions were mde thoroughly fmilir with the Pthwise Evlution System from the Eductionl Testing Service through professionl development experiences provided through the College of Eduction. The items of the form nd their orgniztion into subscles clled domins re shown in Tble 1. Mterils nd Procedures Before interns locted to their respective plcements, they were briefed bout the expecttions for the field experience. Erly childhood mjors nd middle level mjors enrolled in 16-week course for 15 nd 12 semester hours, respectively; secondry mjors enrolled in nine-semester hour course encompssing 12-week internship. Secondry mjors completed n on-cmpus course in public school lw, history nd philosophy of eduction, nd content re reding before beginning their 12-week internship. All interns hd completed substntilly ll of the requirements for their respective mjors except for the internship itself. 14 b
Tble 1 Item Specifiction nd Split-hlf Relibility for Performnce-bsed Assessment of Techer Effectivenss The Formtive Observtion nd Intervention form ws used by cmpus-bsed nd field-bsed supervisors for evlution purposes nd to provide feedbck to interns. For the purposes of this study, we decided to use the form to investigte the fctor structure of effective teching, using dt from 130 interns of the spring semester of 2010. It ws the intent of the supervisory experience to observe ech intern t lest four times while the intern ws teching; this occurred in most but not entirely ll instnces. Prior to this investigtion, previous study utilizing the sme dt hd been done to determine the relibility, vlidity, nd suitbility of the Formtive Observtion nd Intervention form in our ppliction of it. These fcets of the form were believed to be more thn dequte (Womck, Hnn, Woodll, & Cllwy, 2011). Artifct Relibility. The uncorrected split-hlf relibility of the Formtive Observtion nd Intervention form ws 0.976 with 416 usble observtions. The stndrd error of mesurement ws 2.6 points out of 63 possible points on 14
VOLUME 2, ISSUE 1 the entire 21-item instrument. It ppered tht the ssessment ws relible. Artifct vlidity. All items on the form were mpped to the stte s licensing stndrds nd to the Prxis III (Pthwise) ssessments. These mppings were recorded on severl documents tht becme prt of the techer eduction unit s electronic exhibits pursunt to ccredittion by the Ntionl Council for Accredittion for Techer Eduction (NCATE) nd by the Stte. Results Dt from 416 observtions of 130 cndidtes were obtined during the spring semester of 2010. These occurred s fculty or clinicl prctice instructors completed four cycles of evlutions while observing interns in teching situtions. Fctor Loding The principl purpose of our study ws to determine if there were fctor lodings on this mesure of effective teching nd, if principl fctors were found, to determine wht those fctors were by crefully ssigning nmes to them. Procedure FACTOR of the Sttisticl Anlysis System ws used to discover fctors, using the suggested prior communlity estimte of one nd minimum Eigenvlue of one (Htcher & Stepnskie, 1994). Tble 2 Eigenvlues of the Correltion Mtrix (N=416 observtions from 130 teching interns) SAS output indicted tht were likely four fctors within the observtionl dt from the interns tht met these criteri. The fifth fctor ws indicted on Tble 2 to show the reder where the Eigen brek ws. The scree plot ws somewht consonnt with tht finding while indicting the presence of n initil lrge fctor (Figure 1) tht ccounted for 41% of the vrince in teching effectiveness scores. Fctor Detection As Htcher nd Stepnski (1994) nd Ingrm (2011) indicte, interprettion of fctors nd of items correlting with fctors is subjective. This my seem counter to the ppernce of the mthemticl precision of the output of progrm like PROC FACTOR, but reserchers, given few suggestions from the sttisticl literture, re left to dopt their own criteri for fctors nd items. We determined tht we would recognize fctor if it hd n Eigenvlue of t lest one, ppered distinct on the scree plot, nd ccounted for t lest 5% of the vrince. The scree plot (Figure 1) is depiction of the vrince extrcted t ech stge of the fctor nlysis. Htcher nd Stepnski sy The word scree refers to the loose rubble tht lies t the bse of cliff. When performing scree test, you normlly hope tht the scree plot will tke the form of cliff: At the top will be the eigenvlues for the few meningful components, followed by brek (the edge of the cliff). The bottom of the cliff will be like the scree: eigenvlues for the trivil components (p. 473). We determined tht we would recognize n item s being ssocited with fctor if its correltion with fctor reched sttisticl significnce t the.01 level. We plnned to nme fctor in specil considertion of its five gretest correltes (ssuming there would be t lest five), in view of the priori domins from which the items cme, nd in view of the lnguge of the items. The number five ws chosen becuse of the originl minimum of five items per domin on the form. 1 b
Figure 1. Scree Plot of Eigenvlues for Rtings of Techer Intern Performnce 9 ˆ 1 8 ˆ 7 ˆ 6 ˆ E i g e 5 ˆ n v l u 4 ˆ e s 3 ˆ 2 ˆ 2 3 1 ˆ 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21. Item-Fctor Identifiction Items from the Formtive Observtion nd Intervention form were llowed to remin in the fctor structure if they correlted significntly (criticl r=.230, p<.001) with the fctor. Sttisticl significnce is not mndted in fctor-nming, but it is stndrd tht is commonly used. Fifteen items correlted significntly with the first fctor (see Tble 3), fctor tht ccounted for 41% of the totl vrince. Fctor Nming The first fctor ws nmed lesson plnning. In referencing the correltions to the items on the observtion form, two of the top five correltions were with items tht delt very obviously with plnning (A2, A5). B3 (chllenging lerning expecttions) usully occur s result of creful lesson plnning. D2 nd D4 could be considered extensions of plnning in tht plnning promotes efficcy nd sense of cpbility in reching out to prents.
VOLUME 2, ISSUE 1 Tble 3 Rotted Fctor Pttern Using the rotted vrimx fctor pttern, the other three fctors were lso nmed. The number of items correlting significntly with the remining three fctors ws considerbly less. The second fctor correlted significntly with items C3, D1, D2, C5, nd D4. The second fctor ws nmed Techer nd student reflection in considertion of the lnguge of the items bout techer reflection on gols met, initition of modifictions for students needs, nd student higher order thinking. The third fctor ws nmed Firness/sfe environment in view of the lnguge of most of the items contributing to its vrince. The third fctor correlted significntly with items B1, B4, B5, A2, nd D3. Three of these re ddressed in Domin B, Creting Environment for Student Lerning. Domin B ws mesured during the dynmics of ctul instructionl events. The fourth fctor correlted with items D5, D3, D2, A1, nd D4. The fourth fctor ws nmed Professionlism nd efficcy in deference to the predominnt lnguge of the items most ssocited with it on time, professionl ppernce, follows policies... builds professionl reltionships, collbortes... ccepts responsibility, efficcy... reflects on gols met. In n effort to better visulize which items ctully loded with which fctors, we constructed simple incidence tble (Tble 4). It seemed pprent tht items correlted with fctors were spred cross rther thn within wht hd been considered priori to be in different domins. We were ble to see some interreltionships tht mke up the complex tsk clled teching. Plnning, for instnce, touches not only items A1 through A6, but lso on the rpport tht tech- b
ers re ble to build with students (B2), the frming of chllenging lerning expecttions (B3), plnning for physicl sfety (B5), the mking of content comprehensible (C2), nd five other items. Tble 4 Fctors Loded on by Ech Item Discussion Given the nture of the ssessment instrument one designed to ssess effective teching, with its relibility nd vlidity, this study ws not only study on technicl issues, but lso on the nture of effective teching itself. As mentioned erlier, there were four priori ssumptions bout the fctor structure of the instrument tht were tested in this fctor nlysis:
VOLUME 2, ISSUE 1 1. Four fctors would be found, corresponding to the four domins of Pthwise. 2. The items tht mesured these fctors would be locted within the domin structure suggested by Pthwise. 3. The two items tht hd been dded loclly would not lod significntly upon the rest of the fctor structure. 4. Decisions bout the fctor structure would not be bsed hevily upon the two loclly-developed items lone. With regrd to ssumption one, four fctors were found, but they did not nerly correspond to the subscles suggested by our Prxis III-like instrument, the Formtive Observtion nd Intervention form. Regrding ssumption two, the items tht loded most hevily on ech of the four fctors were not ll from the respective domins of the form; rther they were scttered cross severl domins. The first nd lrgest fctor, tht of plnning, hd item lodings from ll four domins. Lesson plnning correlted significntly with 15 of 21 items of our reserch-bsed instrument tht were designed to ssess effective teching. Only in the fourth fctor were most of the five most-correlted items from the domins tht hd been suggested priori. Regrding the third ssumption bout the two loclly developed items ones tht hd not been expected to lod or correlte with the rest of the instrument item A6 s reflection of totl prepredness to tech specific lesson ws t lest significntly correlted to three of the four fctors. The loclly-dded item on Domin D, item D5 bout being on time nd meeting professionl responsibilities, loded on nd ws significntly correlted on fctor four, being the most correlted of the items within the fctor. Thus, speking to the fourth ssumption, while decisions bout the fctor structure ended up being relted to the two loclly-developed items, the dt did not suggest tht these two items were out of plce, compred to the 19 ETS-bsed items. The Vlue of Lesson Plnning Forty-one percent (41%) of the vrince in effective teching in our interns ws ccounted for by lesson plnning. Tht is, before they wlked into clssroom nd uttered the first word of the dy, 41% of student lerning hs lredy been decided by the prepredness or lck thereof of the techer for tht specific moment. Intuitively we in techer eduction hve emphsized to novice techers the importnce of creful nd thorough lesson plnning. With the findings of this study, tht importnce need no longer be one dvnced only by intuition. Lesson plnning s significnt endevor goes beyond just deciding which method or which ctivity to utilize in lesson. There ws little evidence in our findings to promote ny prticulr methodology s pnce for teching ny or ll subjects. Rther, lesson plnning touches the eventul method of ssessment tht students will fce, plnning for sfety in the physicl environment, plnning for firness, plnning for chllenging lerning expecttions nd for higher-order thinking, plnning for effective pcing nd time on tsk, nd more. Techers who re constntly prepred for the next dy, week, nd month of teching find it esy to pproch nd interct with prents. It is esier to cultivte rpport with students when Wht will I be doing next period? is not rel concern. For these nd other resons, the vlue of lesson plnning cn hrdly be overstted. The Vlue of Reflection nd Higher-order Thinking Techer reflection nd student higher order thinking, the second lrgest fctor, ccounted for 6.47% of the totl vrince in techer effectiveness. Reflection enbles techers t ll experience levels to gin much more from their experiences thn just the initil exposure. Our interns re required to write reflections bout the events of ech dy. The vlue of higher order thinking for both the techers nd their students is well estblished in the literture. The Vlue of Firness nd of Sfe-School Environment Firness nd sfe-school environment ccounted for 6% of the vrince in techer effectiveness. Students need to be treted firly by techers nd by other students. Students need to be ssured tht their work will be evluted firly by techers. They lso need to be ssured tht they will not be bullied by clssmtes. Most sttes hve pssed lws during the pst decde to del with bullying. Techers nd dministrtors should do their prt in enforcing these long-overdue lws. Professionlism, Responsibility, nd Efficcy Professionlism ccounted for bout 5% of the vrince in techer effectiveness. At lest two Domin D items loded on ech of the four fctors. Professionlism must be prt of everything tht techer does. Professionlism is ex- b
pressed in the effort level tht techers show in lwys being prepred for clsses, in the prepredness tht techers show in dopting nd implementing clssroom mngement strtegies, in the wys tht techers tret other techers nd dministrtors, nd in the wys tht techers seek interctions with prents. Other Vrince About 41% of the vrince ws not ccounted for by the model. This vrince in teching effectiveness ws scttered mong mny smll ctegories. With the high relibility nd smll error of mesurement, it ws not believed tht mesurement error ws lrge fctor. Mny smll but essentil behviors comprise effective teching. They dd in smll but incrementl wys to the totl mount of student lerning tht tkes plce. Conclusions After yers of utilizing the Formtive Observtion nd Intervention form, this study helps the observer to be ble to hve discussion with interns bout the importnce of plnning. As professionls, we often try to stress this to preservice techers, but without much success. Now we hve number tht we cn plce on wht is relly importnt nd to wht degree plnning is importnt. Tht number is 41% of their success. This informtion cn help to give concrete evidence to students s well s techers how importnt their plnning cn be. Knowing wht res mke rel difference cn lso help with plnning on the prt of the university. It seems crucil to spend time trining our pre-service techers in the skill of plnning. Therefore, it is importnt to spend the time in our courses with specific trining on the importnce nd the methodology in specificlly how to pln for teching. The vlues of techer reflection nd of student higher-order thinking re well estblished in the literture. When techers reflect, they re ble to re-experience lesson mny times over nd to lern from both their successes nd filures. Students bsorb, rerrnge content, nd store it in long-term memory in wys tht re personl nd idiosyncrtic to ech of them. Reflection nd higher-order thinking should continue to be emphsized, regrdless of the grde level of the techers nd students involved. Firness nd sfe-school environment hve risen s significnt fctors especilly in the pst twenty yers. Incidents such s those in Jonesboro, Arknss, Columbine, Colordo, nd Virgini Tech hve given heightened wreness of the need to feel secure. Without these feelings of security, higher-order thinking nd reflection re not likely to occur (Mslow, in Ormrod, 2004, pp. 432-433). In the pst genertion, our society hs become more wre of bullying nd the long-term, negtive effects of bullying. Students need to feel sfe not only from the forces outside of the clssroom, but lso from those tht re within. The fctor of professionlism, responsibility, nd efficcy lies not only in the interctions with other techers nd prents, but goes much deeper. It involves the techer cring bout their profession. It involves the techer tking on the responsibility for their students lerning. Interestingly, without good dily plnning, it is nerly impossible for ny of this to hppen. In conclusion, it ll comes bck to plnning. Without substntil effort nd skill in this re, the intern or techer cnnot effectively estblish clssroom of lerning tht is fir, sfe, elicits higher order thinking, or enbles students to grow in productive mnner. We, s techer eductors, must be prepred to model nd tech these skills to our pre-service techers nd to our interns in order to enhnce their opportunities to succeed nd become effective techers. Recommendtions A recommendtion for future reserch would be to further explore which kinds of plnning seem to enhnce techer effectiveness the most. It is likely tht ll forms of plnning re not eqully productive. References Arknss Deprtment of Eduction. (2009). Schedule for novice techer observtions. Retrieved from http://www. rknssed.org/techers/pdf/im_observtions_0107.pdf. Eductionl Testing Service. (1996). Assessment criteri [nd other Pthwise trining mterils]. New Jersey: ETS.
VOLUME 2, ISSUE 1 Ferguson, G. A. (1976). Sttisticl nlysis in psychology nd eduction. New York: McGrw-Hill, p. 494. Htcher, L., & Stepnski, E. J. (1994). A step-by-step pproch to using the SAS System for univrite nd multivrite sttistics. Cry, N. C: SAS Institute Inc. Hill, T. & Lewicki, P. (2007). Principl components nd fctor nlysis. Retrieved online from http://www.sttsoft.com/ textbook/principl-components-fctor-nlysis/ STATISTICS: Methods nd Applictions. SttSoft, Tuls, OK. Ingrm, P. (n.d) Multivrite sttistics: Fctor nlysis. Retrieved from http://www.socilreserchmethods.net/tutoril/flynn/fctor.htm. Ntionl Council for Accredittion of Techer Eduction. (2000). Plnning instrument (Revised 2002 edition.) Wshington, DC: NCATE. Ntionl Council for Accredittion of Techer Eduction. (2008). Abbrevited plnning instrument for 2008 NCATE stndrds. Wshington, DC: NCATE. Ormrod, J. E. (2004). Humn lerning. Columbus, OH: Person. U. S. Census Bureu. Pope County quickfcts from the U. S. Census Bureu. Retrieved from http://quickfcts.census. gov/qfd/sttes/05/05115.html. Womck, S., Hnn, S., Woodll, & Cllwy, R. (2011, April). Intern performnce in three supervisory models. Arknss Assocition of Colleges of Techer Eduction: Sercy, Arknss. Dr. Sid T. Womck hs been in eduction for 40 yers. He begn teching s bnd director in 1972 in Trinity, Texs, fter completing Bchelor of Music Eduction degree from Abilene Christin College. Although successful s music eductor, he took different direction on grdute degrees in view of position vilbility for music eductors, prticulrly in higher eduction, in the 1970s. After completing his Mster of Eduction degree in elementry eduction nd specil eduction t Sm Houston Stte University in Huntsville, Texs, in 1974, he tught high school specil eduction for yer nd then fourth grde in Mdisonville, Texs, for yer, prepring diversified set of experiences for his eventul college udiences. He completed his Ph.D. in eductionl curriculum nd instruction with concentrtion in elementry eduction from Texs A&M in 1979. He returned to public school teching in the 1980s during turbulent economic times in higher eduction, teching secondry emotionlly disturbed nd lerning disbled students in secondry schools in Texs nd Oklhom. In the 1980s, he returned to grdute school t Sm Houston Stte University to complete certifiction work in eductionl ledership. He hs held certifictes for the principlship in both Oklhom nd Texs. Dr. Womck is now professor of secondry eduction t Arknss Tech University, where he hs been for the pst 26 yers. He hs written over 50 juried rticles nd mde over two dozen refereed presenttions on elementry nd secondry eduction, specil eduction, eductionl psychology (his doctorl minor) nd eductionl ledership. He is bord member for childrens home in Arknss. He is n elder in the church where he ttends. His emil ddress is swomck@tu.edu. Dr. Shellie Hnnh hs been in eduction since 1992. After completing degree in History nd Politicl Science in 1988 nd completing techer eduction certifiction in 1990 from Arknss Tech University, she dded n Arknss English licensure the following yer. Dr. Hnn tught socil studies in the Russellville School district in Arknss for four yers. She lter completed Mster s degree in Physicl Eduction t Arknss Tech in 2002. She begn teching t Arknss Tech in 2006 nd recently completed her doctorte in eductionl ledership with concentrtions in curriculum nd instruction t Orl Roberts University in Tuls, Oklhom, in My 2009. She is recognized leder for helth nd fitness on the Arknss Tech University Cmpus. She directs Tech Fit, fitness center which serves over thousnd students nd fculty; mnges tem of 10-15 employees; nd teches courses in wellness science, physicl eduction, nd physicl fitness. Dr. Hnn performs presenttions t locl, stte, nd ntionl levels tht del with both teching strtegies nd physicl fitness issues. She lso teches fitness clsses for locl fitness center nd is b
ctive in volunteerism in both her community nd church. The emil ddress where she my be reched is shnn@ tu.edu. Dr. Columbus Dvid Bell hs been in eduction for bout 40 yers. He ws n elementry nd kindergrten techer in Northwest Arknss for nine yers before completing his Mster s nd Doctor s degrees in Eduction t the University of Arknss t Fyetteville. He ws fculty member t severl Arknss universities before coming to Arknss Tech University in 1988. He is the licensure officer for ATU. He enjoys teching childrens literture s well s his dministrtive work for Tech. His emil ddress is dbell@tu.edu.