Outcome Based Education Performance Evaluation on Final Year Degree Project

Outcome Based Education Perfmance Evaluation on Final Year Degree Project NORAZLAN HASHIM, HADZLI HASHIM Faculty of Electrical Engineering, Universiti Teknologi MARA, 40450 Shah Alam, MALAYSIA. azlan4477@salam.uitm.edu.my, hadzli0@salam.uitm.edu.my Abstract: - wadays, implementing Outcome Based Education (OBE) to evaluate course outcomes (CO) program outcomes (PO) is a stard practice at the Faculty of Electrical Engineering (FEE), Universiti Teknologi MARA (UiTM). This includes the evaluation of the final year degree project () since is a maj component of the undergraduate degree course in Electrical Engineering. The evaluation of mainly consists of two stages. The first stage involves the evaluation of the technical paper project presentation by a Technical Paper Assessment Panel (TPAP). The second stage involves the evaluation of thesis wk by the respective Project Supervis (PS). These s inconsistence in nature as each stage involves many lecturers from different background of disciplines in the FEE. Furtherme, there were no specific f the grading process lecturers would rely on their experiences, resulting large variance between the senis junis judgments in giving the marks. To overcome such problem, a powerful OBE evaluation tool known as Project Sens Perfmance Evaluation Course Tool (PRO-SPECT) has been designed f evaluating the. The output plots produced by this tool would be used as indicats f Continual Quality Improvement (CQI) recommendations. This paper presents the process of how students being assessed when taking the module by using the PRO-SPECT tool. Key-Wds: - Outcome based education; final year degree project; continuos quality improvement Introduction Outcome Based Education (OBE) is an education system that emphasis on outcomes measurement rather than inputs of curriculum covered. Outcomes may include a range of knowledge, skills attitudes. In der to obtain the desired outcomes, teaching components activities should be well ganized, planned continuously improved [, ]. OBE concept has been applied in many countries ranging from primary schools to universities. In Malaysia, the Engineering Accreditation Council (EAC) of Malaysia has directed that all engineering programs must adopt implement OBE concept beginning 007 [3]. In fact, it has become a maj requirement f any degree program to be accredited by EAC. Faculty of Electrical Engineering (FEE) at the Universiti Teknologi MARA (UiTM) has introduced OBE knowledge amongst its staff since 005. Beginning 007, all degree courses have OBE printed in each of its syllabus. Another wds, every course has their course outcomes (CO) being mapped the FEE targeted program outcomes (PO) [4]. This mapping is known as CO-PO matrix. There eleven Program Outcomes (PO) decided by the FEE as outlined in Table. Table : Program Outcomes F Faculty of Electrical Engineering, Universiti Teknologi MARA Program Outcome PO PO PO3 PO4 PO6 PO7 Attributes Ability to acquire apply science engineering fundamentals. Ability to express ideas effectively, in written al fm. Acquiring in-depth technical knowledge in one me specializations. Ability to identify, fmulate solve engineering problems. Ability to utilize systems approach to design evaluate operational perfmance. Ability to use the, skills, modern engineering tools necessary f engineering practices. Ability to recognize appreciate imptance of ethical stards in professional wk. PO8 Ability to acquire lifelong learning. PO9 PO0 PO Ability to apply managerial entrepreneurship skills. Ability to wk as both an individual in a team on electrical engineering multi-disciplinary projects. Knowledge of contempary issues appreciation of diversity in the wld intellectual as. Students enrolled in this program, expected to acquire these outcomes at the end of their four year of ISSN: 79-46X 5 ISBN: 978-960-474-0-8

studies through various courses offered in the Bachel of Engineering (Hons) Electrical Engineering program EE0. At the FEE, the course assessment activities divided into four components that examinable courses, non-examinable courses, labaty courses final year degree project course. Each of these components has its own customized measurement tools f OBE perfmance as described in Table below [5, 6]. Table : Faculty Of Electrical Engineering OBE Measurement Tools N o 3 4 Name Summative Dynamic Assessment Model OBE Compliance Outcome Based Perfmance f n-exam Courses Labaty Sens Perfmance Evaluation Course Tool Project Sens Perfmance Evaluation Course Tool Code SAMOBEC OPNEC LAB-SPEC PRO-SPEC Fmativ e Activity Test, Assignme nt, Quiz Test, Assignme nt, Quiz, Project Group Related Skill, Rept Wk Progress Summative Activity Final Exam Presentation, Demonstrati on ne Technical Paper, Presentation, Thesis Final Year Degree Project Module The is a ce course designed f the final year degree students to acquire new knowledge experience in project wks not only related to electrical engineering but also to other related technical as. This course has been designed to contribute to the following outcomes;. Solve research problem using appropriate, tools, skills algithms (CO).. Design, analyse evaluate research wks (CO). 3. Present project findings effectively produce technical paper thesis (). This course is expected to provide the students an infmal training on the key of project management such as time management, research planning scheduling, communication skills, problem solving lifelong learning. Apart from that, this course will also help to prep the students necessary skills f pursuing a postgraduate degree in the future. This course required students to complete the project at the end of 8, students will demonstrate their projects to their project superviss there will be a question answer session during the presentation. Students also required to produce a technical paper based on the project. The technical paper presentation will be evaluated by panels from members of the faculty. At the end of the 8 th, students required to write a technical paper in enhancing students capability in technical writing based on their technical wks. This activity would assist the students to rept their wk professionally accding to a specific given by the Institute of Electrical Electronics Engineers (IEEE). Upon the submission of technical paper, students required to present the technical paper to the Technical Paper Assessment Panel (TPAP) as scheduled by the Codinat. The main objective of this activity is to give opptunity to the students to defense present their wk, findings achievements of the project. Apart from that, the presentation would also assist the TPAP in clarifying any materials written in the technical paper from the students. On top of that, students will be evaluated based on their communication presentation skills. Each student is given approximately 0-5 minutes to do their presentation followed by a 5 minutes question answer session. Since the duration f the presentation is nmally sht, students expected to ganize their presentations effectively. In, students also evaluated based on their wk. Therefe, students expected to meet their respective superviss regularly to present their. In addition, students required to maintain their logbooks accdingly. Only students satisfacty allowed to submit their technical papers thesis. Students also required to present their complete project wk in the fm of well-structured rept (thesis). The motive is to enhance students writing skill it is popularly considered as the ultimate task in an undergraduate degree program. Both wk project rept will be evaluated by the project supervis. As a summarization, the evaluation of consists of four maj sections as depicted in Table 3 below. Table 3: Evaluation f Sections Marks Evaluat. Technical Paper Evaluation 5 TPAP. Project Presentation 5 TPAP 3. Student Progress Evaluation 40 PS 4. Final Rept Evaluation (Thesis) 0 PS The codinat would then compile all the marks collected from the Technical Paper Assessment Panels Project Superviss transfmed them into grades. ISSN: 79-46X 6 ISBN: 978-960-474-0-8

3 PRO-SPEC Evaluation Tool The OBE evaluation tool has been designed f evaluating module. This measurement tool, known as Project Sens Perfmance Evaluation Course Tool (PRO-SPECT) shown as in Figure, is an Excel based designed softw. It has four evaluation sections; Technical Paper Evaluation, Presentation Evaluation, Student Progress Evaluation Thesis Evaluation. In evaluating students fairly, lecturers being provided by rubrics marking scheme f each topic in the sections above. these raw marks plus inion on the appropriate CO PO will be used as the input when using the PRO-SPECT tool. The COs POs ad by the four sections above tabulated in Table 4. It can be observed that, there is a direct one-to-one mapping between each CO its respective PO. The system tool finally will produce perfmance plots that show students POs achievement sces. In addition, lecturers can also analyse other plots such as COs POs density, as well as students population density in achieving the sces. these plots can be used by the lecturer to prep a Continual Quality Improvement (CQI) rept f the labaty module. Fig. : PRO-SPECT Introducty Page PROJECT PRESENT ATION EVALUAT ION PROGRES S EVALUAT ION THESIS EVALUAT ION Conclusion: Students should be able to conclude the findings in addressing the objective of the project & Recommendation f future wk PO Fmat: Written accding to PO CO CO CO CO Engagement: Appearance, gesture, voice & eye Presentation Skills: Suitable Tone of Voice, Fluent English usage, Effective Use of Presentation Aids, Convincing Content: The presentation slides should consist the followings: Introduction, Methodology/Project wk, Discussion, Conclusion Recommendation Question Answers: Ability to answer questions convincingly. Attendance: Ability to frequently meet supervis Creativity: Ability to gather inion resources f the given problem Wk : Ability to use recd any wk in a logbook f a given timeline Demonstration finding of results: Ability to demonstrate analyse results appropriate reasonable explanation Abstract: Students should be able to briefly summarize what has been done, also demonstrate the findings of the project Introduction : Background of Study, Problem Statement, Problem Identification, Significance of the study, Objective, Scope of Wk & Thesis Organization Literature Review: Students should be able to review the in the scope of study & Students should also be able to perfm on previous wks Methodology/Project Wk: Student should include the algithm, flow charts pseudo codes of the programming codes OR/AND; Students should include the hardw design, block diagram, appropriate circuitry towards achieving the project outcomes Discussion: Students should exhibit the significant results of the project, Students should be able to discus analyse the results of the project Conclusion: Students should be able to conclude the findings in addressing the objective of the project References: Students should write the in accdance to the specific (i.e. IEEE ) Others: Writing Style, Grammar & Compliance to the stard/guideline PO PO PO PO6 PO PO PO PO PO PO PO PO PO Table 4: COs POs Specifications f Sections CO CO Attributes TECHNIC AL PAPER EVALUAT ION Abstract: Objective(s), Scope of Study, Methodology & Findings Introduction: Overview of Study, Problem Statement, Problem Identification, Significance of the Study, Objective Scope of Study Methodology: Algithm, flow charts pseudo codes of the programming codes OR/AND, hardw design, block diagram, appropriate circuitry towards achieving the project outcomes Discussion: Exhibit the significant results of the project, Discus analyse the results of the project Addr essin g PO PO PO PO PO The first step of using the tool is to identify the CO-PO mapping of respect to Table 4. The CO-PO mapping of the respective evaluation sections will be printed in Table 5 Table 8 automatically. Only the strongest PO is identified f each CO. These mapping enables the distribution of the CO ad by each evaluation sections eventually, the relationship between COs POs that being ad can then be observed. ISSN: 79-46X 7 ISBN: 978-960-474-0-8

Table 5: CO-PO Mapping f (TPE Section) Coding TPE TECHNICAL PAPER EVALUATION (TPE) TPE TPE 3 TPE 4 TPE 5 TPE 6 Marks 0.00 5.00 0.00 30.00 0.00 5.00 Course Outcomes Programme Outcomes (Please Refer Course CO-PO Matrix) The strongest PO will be selected AUTOMATICALLY PO PO PO3 PO4 PO6 PO7 PO8 PO9 PO0 PO Table 6: CO-PO Mapping f ( Section) Coding PROJECT PRESENTATION EVALUATION () 3 4 5 6 7 CO 8 Marks 5.00 0.0 5.00 5.00 5.00 0.0 5.00 5.00 Course Outcomes CO Programme Outcomes (Please Refer Course CO-PO Matrix) The strongest PO will be selected AUTOMATICALLY PO PO PO3 PO4 PO6 PO7 PO8 PO9 PO0 PO Table 7: CO-PO Mapping f (PE Section) Coding PE CO PROGRESS EVALUATION (PE) PE CO PE 3 PE 4 Marks 0.00 5.00 0.00 5.00 Course Outcomes CO CO Programme Outcomes (Please Refer Course CO-PO Matrix) The strongest PO will be selected AUTOMATICALLY PO PO PO3 PO4 PO6 PO7 PO8 PO9 PO0 PO Table 8: CO-PO Mapping f ( Section ) Programme Outcomes (Please Refer Course CO-PO Matrix) The strongest PO will be selected AUTOMATICALLY Coding FINAL REPORT EVALUATION () 3 4 CO 5 6 7 8 Marks 0.0 0.0 0.0 5.0 5.0 5.0 5.0 0.0 Course Outcomes CO PO PO PO3 PO4 PO6 PO7 PO8 PO9 PO0 PO 4 Perfmance Analysis Once the CO-PO mapping is done, students marks f the evaluation sections ready to be entered processed f any output measurements. Next, the system tool will produce perfmance plots that show the CO density as well as PO density as depicted in Figure. As can be seen from Figure (a), this is concerning on 3 COs have the highest density that is 87.93%. This indicates that students required to pay me attention to that is to present project findings effectively to produce technical paper thesis. The density f CO which is focusing on solving research problem using appropriate, tools, skills algithms is about 3.45%. Whereas, the density f CO which is focusing on designing, analysing evaluating research wks is about 8.6%. of these COs mapped to three POs respectively as depicted in Figure (b). Since each CO addressing one PO, thus all the three POs have a weight of 00%. ISSN: 79-46X 8 ISBN: 978-960-474-0-8

(a) CO Density (b) PO Density Fig. : Distributions of COs POs f Guidelines f giving the marks can be referred to the rubrics table shown in Table 9. These rubrics mapping of students perfmance exercise will actually allow TPAP PS to justify any marks given to their students. These marks will be filled in a customized evaluation fm provided by the codinat at the end of the 8 th. Since PRO-SPECT produces output measurement of POs indirectly, the COs achievement by the students, therefe the processing engine in the tool is designed specifically f these purposes. The algithm is being fmulated to map compute all input marks its respective PO. The final measurement sce is computed by nmalizing the actual student s earned point the total maximum possible point. Equation describes the calculation of a PO sce respectively. During the process of calculating a specific PO sce, all other non- POs COs disabled by labeling them as logic 0. Alternatively, only the interested PO is labeled as logic. In that case, only the respective student s mark f this PO is considered in the calculation process. Components TECHNICAL PAPER EVALUATIO N Table 9: Rubrics Evaluation Fm f (Very Weak) t do not meet (Weak) Only (Moderat e) Only (Strong) Only 4 but not (Very Strong) PROJECT PRESENTATI ON PROGRESS EVALUATIO N THESIS EVALUATIO N project s objective conclusion on the achieveme nt of project objectives, recommen dation of future wk Wrong paper structure not fmally, no facial eye is fullfilled is fullfilled answers Meet less than 3 Too dependent not creative logbook no met t Literature review ir to study out Wrong paper structure but partially not fmally, satisfact y facial eye is is Answers not related to questions Meet me than 3 than 5 times Dependent but show some creativity logbook po Only results demonstra ted Only Explain previous studies, but no s on pros cons paper structure me than ting fmally, no facial eye Only is Only is Answers related to questions po points Meet me than 5 than 7 times Independe nt, show some creativity Logbook maintaine d po demonstra ted analysed critically, but no explanatio n of results Only Explain previous studies, insufficien t s on pros cons crect but only paper structure less equal to ting fmally, regular facial eye is is Answers related to questions good points Meet me than 9 than 0 times Independe nt creative Logbook maintaine d good demonstra ted analysed critically inaccurate explanatio n of results Only 4 Explain previous studies, good s on pros cons. explanatio n of the need of study at crect paper Fmat fmally, consistent facial /eye Good of ideas, Very convincin g Meet me than 0 Highly independe nt, creative can wk minimum supervisio n Logbook maintaine d advanced demonstra ted analysed critically accurate explanatio n of results Explain previous studies related good s on pros cons, finally explain the need of the ISSN: 79-46X 9 ISBN: 978-960-474-0-8

block diagram there is no s do not meet project s objective conclusion on the achieveme nt of project objectives, recommen dation of future wk in increct Very frequently used wds me than 30 al errs. Wrong observed does not follow the at all block diagram. The s not in sequence, illogical, incomplet e unclear out Me than 5 in increct Very seldom used wds me than 0 al err equal to 30 al err. Wrong observed does not follow the at all There is block diagram the s partially in sequence, logical, complete partially clear Me than 3 less equal to 5 in increct ly used wds me than 0 al err equal to 0 al errs. observed follow the the end There is block diagram the s in sequence, logical, complete partially clear crect but only Me than less equal to 3 in increct ly used wds exists good transitions between statements. Has me than 5 al errs equal to 0 al errs. observed follow the study There is block diagram s in sequence, logical, complete very clear crect in crect Good variation in using wds good transitions coherence between statements. Has less than 5 al err proper complianc e to T C C SM PO i C C SM POi - - totalcomponent - - Cj Cj C SM x PO j C i SM j PO student i % PO x 00 j i,,3... T totalcomponent C C RM PO C j Cj i RM x POi C C RM PO j i - - - - C j C j RM PO i x 00 Where; SM = student s marks RM = total marks PO = program outcome (logic 0 ) C = assessment component Raw marks from the evaluation fm submitted by the TPAPs PSs compiled data entry will be done by the codinat. Once the students marks entered, measurement of their POs sce will be automatically produced. Simultaneously, an average sce f the respective POs will be computed which represent the overall perfmance of the students taking this. of these sces displayed in terms of plot thus, conclusion can be derived from them. An example of this plot is shown in Figure 3 below. Fig. 3: Distributions of PO Average Sce f As illustrated in Figure 3, it can be seen that all POs can be considered as strongly being achieved where the sce me than 70%. which is related to the system approach has the highest sce, indicating the students managed to utilize systems approach to design evaluate operational perfmance. PO which is focusing on expressing ideas effectively in written al fm has the lowest PO sce that is around 7.55%. This is due to some of the students did not perfm well during the project presentation po thesis writing. The next phase involves transfming the PO sces into a qualitative ranking () ISSN: 79-46X 0 ISBN: 978-960-474-0-8

level by using the guide line table given by the FEE. Table 0 refers also as the Key Perfmance Index (KPI) to be achieved where a sce of me than 65% will be defined as very strong. Actually, this discretion table is iginated from the applied conventional grade point average system where anything less than 50% is considered to be non-perfming below grade C, a sce me equal than 65% reflects a grade B [7]. Sustaining a strong ranking B f every course will eventually result in the students to at least graduate w i t h a n u p p e r s e c o n d c l a s s h o n o r s. Besides that, perfmance of the students in can always be easily monited, tracked compd r egu l arl y b et w ee n on e g ro up t o anot her. Fig. 4: Distributions of PO Average Ranking f Table 0: Key Perfmance Index Ranking PO Sce (%) Rank Level Description Col Code 0-49 Weak Red 50-64 Moderate Yellow 65-00 3 Strong Green Figure 4 represents the PO average sce in terms of ranking level of achievement. As shown in the previous figure, the three POs ad by this course had achieved sce me than 70% which is very strong (level 3). Therefe, this batch of students has shown strong attributes in the identified POs. PRO-SPECT can also provide avenue f the lecturer to dissect his/her students density perfmance so that future improvement can be made during delivery in the. Such indicat is described in Figure 5, where it can be observed that almost 77% of students have strongly achieved PO whereby only 54% of students have achieved PO6 which is related to the ability to use the, skills modern engineering tools f engineering practices. About 4% of students found to be weak in PO6 about 3% weak in. Fig. 5: Students Density f Different Ranking Level 5 Continuous Quality improvement As this course is considered as a maj component of the undergraduate degree course in Electrical Engineering, students expected to be familiar all the imptant to be practiced when doing the. From all the plots observed, at this stage it can be considered that all POs that ad by this course had achieved the target. However, there is still room f improvement especially to increase the students perfmance f PO6 sce. The recommendation f improvement includes the followings:. Increase the understing of the concept of engineering technique skill in solving the problems so that students can apply the knowledge in solving their problems.. Encourage motivate students to use/exple modern engineering tools in their. ISSN: 79-46X ISBN: 978-960-474-0-8

6 Conclusion The process of evaluating the outcomes of Final Year Degree Project () module in FEE, UiTM used to be very complicated as the module consists of four evaluation sections. Thus, a user friendly reliable suppting tool known as PRO-SPECT has been developed to facilitate the lecturers/ codinats to do the evaluations. The system offers a systematic ways in evaluating perfmance that consists of Technical Paper Evaluation, Project Presentation Evaluation, Student Progress Evaluation Final Rept (Thesis) Evaluation. The system will use raw marks gained by students compute their measurement sce of the respective POs. The system outputs in the fm of various plots that can provide indicats to the lecturers/ codinats f recommending further improvement. This paper has described the step by step algithm used by the PRO-SPECT to evaluate in FEE, UiTM. Students raw marks from the assessments activities during the December 009-April 00 session were used as inputs f the system. Outputs plots of average sce ranking of achieved POs as well as the students density f the three different ranking levels shown. These plots can be used analysed thoughly by the respective lecturer/ codinat later make recommendations to be implemented f Continuous Quality Improvement (CQI) exercise. Degree Program, Universiti Teknologi MARA. References: [] W. G. Spady, Organizing f : The basis of Authentic Restructing Refm, Educational Leadership, Vol. 46,., 998, pp. 4-8. [] W. G. Spady, K.J. Marshall, Beyond Traditional Outcome-Based Education, Educational Leadership, Vol. 49,., Oct 99, pp. 67-7. [3] Ministry of Education Malaysia. (006). The Future of Engineering Education in Malaysia. United Mind. [4] Faculty of Electrical Engineering (007), Program Self Assessment Rept, Electrical Engineering Programme, Universiti Teknologi MARA, Malaysia. [5] M. M. Kamal, H. Hashim W. Mans, Summative Assessment Of Outcome-Based Education In Electrical Engineering, in Proc. of the 4 th Int. Conf. on University Learning Teaching (INCULT 008), Shah Alam Selang, 0- Oct. 008. [6] W. Mans, H. Hashim, S.A. Che Abdullah, M.U. Kamaluddin, M.F. Abdul Latip, A.I. Mohd Yassin, T.K. Abdul Rahman, Z. Zakaria, M. Md Kamal, Preliminary on the Implementation of Outcome-Based Education on the n-examinable Computer Engineering Modules, 38 th ASEE/IEEE Frontiers in Education Conference, 008, pp. S4B-0-S4B5. [7] Academic Affair Division. (009). Academic Regulation, Diploma ISSN: 79-46X ISBN: 978-960-474-0-8