An Electronic Tool for Measuring Learning and Teaching Performance of an Engineering Class

Transcription

1 A Electroic Tool for Measurig Learig ad Teachig Perforace of a Egieerig Class T.H. Nguye, Ph.D., P.E. Abstract Creatig a egieerig course to eet the predefied learig objectives requires a appropriate ad effective assesset tool beig able to deterie the level of achieveet of the learig objectives. The basic tools that are usually used for assessig kowledge of studets i a course iclude hoework, quizzes, tests/exas, lab reports, oral presetatios, ad projects. These basic tools ay help to verify what studets are able to do after takig the course; however, they are hardly used to deterie how well the course achieveet addresses each of the selected learig objectives. The priary purpose of this research is to develop a coputer tool for quatifyig the learig ad teachig perforace of a egieerig course with respect to the selected learig objectives as well as idetifyig ecessary actios to be take for iprovig the course quality. For this particular project, the ABET (Accreditatio Board of Egieerig ad Techology) Egieerig Criteria 2000 (also kow as eleve ABET Egieerig Outcoes 3A-K) were used as the learig objectives. Idex Ters ABET Egieerig Criteria, Egieerig Educatio, Learig Assesset, Learig Perforace, Teachig Perforace. I. INTRODUCTION The ABET (Accreditatio Board of Egieerig ad Techology) Egieerig Criteria 2000 (also kow as eleve ABET Egieerig Outcoes 3A-K) were itroduced i the iddle 1990s ad have bee used to evaluate all Aerica egieerig progras sice the [1]. Sice the ew ABET accreditatio syste was first itroduced, researchers/educators have extesively discussed various ethods to assess Outcoes 3a-3k, as show i Table 1. Obviously, the potetial of the ew syste to iprove istructio depeds strogly o how well egieerig faculty uderstad it ad achieve its specified outcoes. Usually, the faculty goes through three ajor activities, whe developig a course to coply with the ABET egieerig criteria, icludig: plaig, istructio, ad assesset/evaluatio. Aog these activities, assesset usually plays a iportat role i the success of the course; as it deteries the level of achieveet of the ABET outcoes, which i tur idetifies ecessary actios to be take for iprovig the course quality. The basic tools that are usually used for assessig kowledge of studets i a course iclude hoework, quizzes, tests/exas, lab reports, oral presetatios, ad projects. These basic tools ay help to verify what studets are able to do after takig the course; however, they are hardly used to deterie how well the course achieveet addresses each of the selected learig objectives. The priary purpose of this paper is to describe a electroic tool that was developed to easure the level of achieveet of the ABET egieerig outcoes of a civil egieerig ad costructio course at Califoria State Uiversity at Log Beach. The paper is orgaized as follows: (1) the overall assesset process for a egieerig class is briefly described; (2) the basic cocept of quality fuctio deployet (QFD) is explaied; ad (3) the developet of the proposed assesset tool usig QFD ethod is preseted. Fidigs ad results are discussed, followed by coclusios. Table 1. ABET Criterio 3 Outcoes A. a ability to apply the kowledge of atheatics, sciece, ad egieerig. B. a ability to desig ad coduct experiets, as well as to aalyze ad iterpret data. C. a ability to desig a syste, copoet, or process to eet desired eeds. D. a ability to fuctio o ulti-discipliary teas. E. a ability to idetify, forulate, ad solve egieerig probles. F. a uderstadig of professioal ad ethical resposibility. G. a ability to couicate effectively. H. the broad educatio ecessary to uderstad the ipact of egieerig solutios i a global ad societal cotext. I. a recogitio of the eed for, ad a ability to egage i lifelog learig. J. a deostrated kowledge of coteporary issues. K. a ability to use the techiques, skills, ad oder egieerig tools ecessary for egieerig practice. Mauscript received July, 11, T.H. Nguye is with the Califoria State Uiversity at Log Beach, Log Beach, CA USA (Telephoe: ; fax: ; e-ail: thguye@ csulb.edu). II. ASSESSMENT OF STUDENT LEARNING The basic startig poit for assessig studet learig begis with askig two basic questios: 1.) what should studets kow ad be able to do at the ed of the seester?, ad

2 2.) what evidece will idicate that they have reached these goals? The basic tools that are ofte used for assessig kowledge i a particular course are the studet works such as hoework, quizzes, tests/exas, ad projects/labs reports. These basic tools ay help to aswer the two basic questios above, however, they are hardly used to deterie how well the course achieveet addresses each of the ABET Criterio 3 outcoes. I additio, the teachig perforace of istructor usually ipacts the success of a class ad should be take ito cosideratio whe assessig the class achieveet. This leads to a eed for a tool that is desiged to assess both teachig ad learig perforace with respect to predefied learig objectives. The assesset tool will help the istructor to collect ad aalyze data cotaiig iforatio likig studet expectatios (i.e. learig objectives or ABET criteria) to a set of teachig perforace etrics that the istructor ca the easure ad cotrol. I this particular study, the studet expectatios or learig objectives iclude the eleve ABET Egieerig Criteria 3A-K (see Table 1), whereas the teachig perforace etrics represet the critical factors that ost affect the istructor s perforace. The teachig perforace factors, based upo results of previous research i [2], iclude the followig factors: 1. Couicatio skill: The ability of the istructor to couicate effectively 2. Studet focus: The degree of iportace that a istructor places o studet relatioships ad studet satisfactio ad the degree of kowledge about studet eeds or learig objectives. Exaples: displayig a persoal iterest i studets ad their learig, ispirig studets to set ad achieve goals which really challege the, ad ivolvig studets i hads-o projects such as research, case studies, or real life activities. 3.Kowledge/Expertise: The istructor s kowledge or expertise about the subject beig taught. Exaples: Capability of akig it clear how each topic fit i the course, explaiig course aterial clearly ad cocisely, ad relatig course aterial to real life situatios 4. Attitude about ehacig studet learig: The degree of ecourageet by the istructor so as to axiize learig of all studets with varyig learig styles ad abilities. Exaples: fidig ways to help studets aswer their ow questios, forig teas or discussio groups to facilitate learig, ecouragig studets to use ultiple resources (e.g. data baks, library holdigs, outside experts) to iprove uderstadig, ad providig tiely ad frequet feedback o tests, reports, projects, etc. to help studets iprove. 5. Iteractio with studets: The degree of effectiveess of the istructor to iteract with studets i class or outside of class (office visits, phoe calls, eails, etc.). Exaples: Askig studets to share ideas ad experieces with others whose backgrouds ad viewpoits differ fro their ow, itroducig stiulatig ideas about the subject, ad stiulatig studets to itellectual effort beyod that required by ost courses. 6. Teachig perforace iproveet: The attitude of the istructor about cotiuously iprovig teachig perforace. Exaples: idetifyig ad evaluatig studet satisfactio ad class perforace results to take appropriate actios for iprovig teachig perforace. These teachig perforace factors greatly ipact the level of achieveet of the class with respect to the learig objectives (i.e. ABET outcoes). Additioally, oe teachig perforace factor ay have a greater ipact tha others ad the differet degrees of ipact aog the factors, i this study, are referred to as stregths of relatioship betwee the teachig perforace factors ad the learig objectives. These stregths of relatioship should be take ito cosideratio as easurig the learig ad teachig perforace of a egieerig class. A appropriate assesset ethod for such a easurig task was selected for this particular study, i.e. Quality Fuctio Deployet (QFD), which is briefly preseted i the followig sectio. III. QUALITY FUNCTION DEPLOYMENT (QFD) QFD, also kow as the House of Quality, is defied as a structured ethodology ad atheatical tool used to idetify ad quatify custoers' requireets ad traslate the ito key critical paraeters that i tur help a copay to prioritize actios to iprove their product or service to eet custoers' expectatios [3]. I a educatio cotext, the custoers represet studets, the service refers to teachig or learig perforace, ad the copay represets the istructor teachig the course. I a busiess cotext, QFD is used for traslatig the voice of custoer through the various phases of project or service plaig, desigig, ad aufacturig ito a fial product. I this educatio study, the QFD ca be used for traslatig the voice of the studet through the three stages of the course plaig, teachig, ad assesset/evaluatio ito the learig objectives. I a typical QFD applicatio, a copay or istructor creates ad aalyzes a data atrix likig custoer or studet eeds (i.e. learig objectives) to a set of service or teachig etrics that the copay or istructor ca the easure ad cotrol. For further details of a QFD applicatio, readers are referred to [4] ad [5]. I this study, the basic QFD process has bee extracted ad odified to develop the process of easurig the level of learig perforace of a egieerig class. Basically, the assesset process iclude the followig steps: first, the iportat weight of each learig objective as well as each teachig perforace factor is deteried; ext, the actual level of achieveet with respect to each learig objective ad teachig perforace factor is collected; the, the stregth of relatioships betwee the learig objectives ad teachig perforace factors are established; ad fially, the learig perforace idex ca be calculated. The learig perforace idex is defied as the ratio of the actual level of learig perforace to the expected or axiu level of learig perforace. I geeral, the process of evaluatig the learig perforace of a class ivolves five ajor steps as follows: 1. Collect data regardig iportat weights, the actual status, ad stregth of relatioships of learig objectives ad teachig perforace factors.

3 2. Calculate poit scores i actual coditio for each teachig perforace factor with respect to a particular ABET criterio or learig objective by usig the followig forula. (W i x AW i ) + (H j x AH j ) Actual S ij = x (IR ij ) 2 Where: i = 1, 2,, (uber of learig objectives) j = 1, 2,, (uber of teachig perforace factors) Actual S ij = poit scores i actual coditio for the teachig perforace factor j with respect to the ABET outcoe i AW i = the actual status of ABET outcoe i AH j = the actual status of teachig perforace factor j W i = the oralized weight of iportace of ABET outcoe i H j = the oralized weight of iportace of teachig perforace factor j IR ij = the stregth of the relatioships betwee teachig perforace factor j ad learig objective or ABET outcoe i. Therefore, the actual level of learig perforace of the class with respect to ABET outcoe i is give by Actual LP i = ΣActual S ij j =1 for 1 j Siilarly, the actual level of learig perforace of the class with respect to teachig perforace factor j is coputed as follows: Actual LP j = ΣActual S ij i =1 for 1 i The actual level of learig/teachig perforace (Actual LP) for the class is calculated as follows: Actual LP = Σ Actual LP i = Σ Actual LP j (1) i =1 j =1 3. Calculate poit scores i axiu (expected) coditio for each teachig perforace factor with respect to a particular ABET criterio or learig objective by usig the followig forula. Max S ij = (W i x MW i ) + (H j x MH j ) 2 x (IR ij ) Where: i = 1, 2,, (uber of learig objectives) j = 1, 2,, (uber of teachig perforace factors) Max S ij = poit scores i axiu coditio for the teachig perforace factor j with respect to the ABET outcoe i MW i = the ax status of ABET outcoe i MH j = the ax status of teachig perforace factor j W i = the oralized weight of iportace of ABET outcoe i H j = the oralized weight of iportace of teachig perforace factor j IR ij = the stregth of the relatioships betwee teachig perforace factor j ad learig objective or ABET outcoe i. Therefore, the axiu level of learig perforace of the class with respect to ABET outcoe i is give by Max LP i = ΣMax S ij for 1 j j =1 Siilarly, the axiu level of learig perforace of the class with respect to teachig perforace factor j is coputed as follows: Max LP j = ΣMax S i =1 ij for 1 i The axiu level of learig/teachig perforace (Max LP) for the class is calculated as follows: Max LP = Σ Max LP i = Σ Max LP j (2) i =1 j =1 4. Calculate the overall perforace idex (PIdex) of the class usig equatios (1) ad (2) as follows: PIdex = (Actual LP/Max LP)x100% (3) I additio, the perforace idex for a particular learig objective i (PIdex i ) or teachig perforace factor j (PIdex j ) ca be coputed as follows: PIdex i = (Actual LP i /Max LP i )x100% for 1 i (4) PIdex j = (Actual LPj/Max LP j )x100% for 1 j (5) 5. Idetify the learig objectives that have low perforace idexes. It is obvious that the desirable learig perforace idex should be as close to 100% as possible. Cosequetly, the learig or teachig perforace factors havig lower perforace idexes should be well take care of for iproveet. IV. DEVELOPMENT OF THE PROPOSED ASSESSMENT TOOL The proposed assesset tool was developed through three steps. First, based upo the QFD cocept o how to deterie

4 the service quality perforace of a orgaizatio, the reasoig of logics (or algoriths) for calculatig the learig perforace idex were established. Secod, the data structure of the QFD odel was used to develop the user iterface for the proposed coputer assesset tool. Basically, the user will be propted to eter the iforatio ecessary for calculatig the expected level ad the actual level of learig/teachig perforace with respect to the ABET criteria. Fially, the assessig algoriths ad the user iterface for the proposed assesset tool were ipleeted ito a coputer platfor. Microsoft Visual Basic 6.0 was used for the research ipleetatio sice it is widely used software progras ad the developet of the prototype ca be easily beta tested ad deostrated to users. The proposed assesset tool was desiged to easure the overall perforace idex of a egieerig class ad was aed PIdex (Perforace Idex). PIdex was used to evaluate learig/teachig perforace of several classes at the Departet of Civil Egieerig ad Costructio Egieerig Maageet at Califoria State Uiversity, Log Beach (CSULB). Below is a exaple iput ad output of PIdex for a egieerig class: CE 130 Surveyig ad Mappig at CSULB. The assesset process i PIdex ca be suarized as follows: Third, the grades of studet works (i.e. hoework, quizzes, tests/exas, lab reports, projects) that were averaged as percetages are etered. Figure 4 shows a exaple of iput data about the level of achieveet of three studet works with respect to the learig objectives A, B, C, D, E, ad K, which is easured as percetages. The zero percetage for learig objective D idicates that the hoework had o relevace with this learig objective. First, the istructor is propted to defie the learig objectives of the class. I this exaple, ABET Egieerig Criteria A, B, C, D, E, ad K were selected (see Figure 1). Figure 2. Iportace Weights of Learig Objectives Figure 1. Defie Learig Objectives Secod, the data about the iportace weights of the learig objectives as well as the teachig perforace factors are iput (see Figures 2 ad 3). These iportace weights were reported fro studets ad seior professors. Also, the stregths of relatioship betwee the teachig perforace factors ad the learig objectives which were give by seior professors are etered. Figure 3. Iportace Weights of Teachig Perforace Factors

5 Fially, the overall learig perforace idex of the class ca be deteried usig equatios (1), (2), ad (3) ad is preseted i the assesset report (Figure 9). I the fial assesset report, the learig objectives or teachig perforace factors that have low perforace idexes are displayed so that the istructor is aware of the weakess ad should take appropriate actios for iprovig quality of future classes. I additio, the perforace idex of the class with respect to each learig objectives (i.e. ABET criteria) as well as each teachig perforace factor ca also be coputed by usig the equatios (4) ad (5), respectively, as show i Figures 6 ad 7. Figure 8 shows the level of achieveet of studet works with respect to the learig objectives. Teachig Perforace Factors Perforace Idex (%) Figure 6. PIdex v.s. Teachig Perforace Factors Figure 4. Iput Data for Studet Works Figure 7. Class Work PIdex v.s. Learig Objectives Figure 5. PIdex v.s. Learig Objectives V. CONCLUSION The assesset of a egieerig course with respect to studet expectatios or learig objectives is a ecessary activity for ehacig the course quality. While traditioal assesset tools such as hoework, quizzes, tests, exas, lab

6 reports, oral presetatios, ad projects ay be used as evideces of the overall studet achieveets with respect to the learig objectives, they do ot address the level of perforace of each idividual learig objective as well as the istructor s teachig perforace. This study ade use of ABET Egieerig 3A-K Outcoes as learig objectives of a egieerig class to be easured. Also, the ost critical factors that ipact teachig perforace of a istructor were evaluated together with the learig objectives. These factors iclude couicatio skill, studet focus, kowledge/expertise, attitude about ehacig studet learig, iteractio with studets, ad teachig perforace iproveet. I this paper, a electroic assesset tool was developed i the coputer platfor of Microsoft Visual Basic 6.0. The tool ca be used to easure the learig perforace idex as the level of the achieveet with respect to the learig objectives of a egieerig class, which idicate the stregth as well as the weakess of the class so that appropriate actios will be take for ehacig quality of future classes. The assesset ethod eploys the cocept of quality fuctio deployet (QFD), a techique to easure the custoer service quality of a orgaizatio, ad applies it to acadeia by likig studets expectatios (i.e. learig objectives) ad teachig perforace of a istructor. The assesset ethod ca be used as a teplate for developig siilar tools to easure learig ad teachig perforace of ay course other tha egieerig courses. Figure 8. Fial Assesset Report ACKNOWLEDGMENT The author gratefully ackowledges the support of the Scholarly ad Creative Activity Coittee progra at Califoria State Uiversity Log Beach for fudig this project. REFERENCES [1] Felder, M. R. ad Bret, R. (2003), Desigig ad Teachig Courses to Satisfy the ABET Egieerig Criteria. ASEE Joural of Egieerig Educatio, Vol. 92, No. 1, pp [2] T.H. Nguye, C. McItyre, ad M. Diab. (2005), Measurig Learig Perforace of a Egieerig Capstoe Course with respect to the ABET Egieerig Criteria Proceedigs of the 2005 ASEE North Midwest Regioal Coferece, Brookigs, SD, Oct , [3] Hoyle, D. (1998), ISO 9000 Quality Systes Hadbook, 3rd Editio, Butterworth-Heiea Ltd., Bosto, MA. [4] Akao, Y. (1990), Quality Fuctio Deployet: Itegratig Custoer Requireets ito Product Desig, Productivity Press, Cabridge, MA. [5] Akao, Y., Mizuo, S. ad Ishihara, K. (1994), QFD; the Custoer-Drive Approach to Quality Plaig ad Deployet, Asia Productivity Orgaizatio, Tokyo.