Outcome-based Education The Assessment of Programme Educational Objectives for an Engineering Undergraduate Degree

Transcription

1 CASE STUDY Outcome-based Education The Assessment of Programme Educational Objectives for an Engineering Undergraduate Degree K.Y. Tshai, J.-H. Ho, E.H. Yap & H.K. Ng Department of Mechanical, Materials and Manufacturing Engineering, University of Nottingham Malaysia Campus, Semenyih, Selangor, Malaysia Corresponding author: K.Y. Tshai, Department of Mechanical, Materials and Manufacturing Engineering, University of Nottingham Malaysia Campus, Jalan Broga, Semenyih, Selangor, Malaysia Phone: +60 (3) Abstract Outcome-based Education (OBE) emphasises on two main components in terms of student achievement in an academic programme. One is the Programme Outcomes (POs) which is measured at the point of graduation, and the other, the Programme Educational Objectives (PEOs) is assessed over a longer period of time (around 4 5 years) after graduation. This study focuses on the establishment of a systematic procedure adopted by the Department of Mechanical, Materials and Manufacturing Engineering at the University of Nottingham Malaysia Campus (UNMC) to formulate PEOs assessment criteria with integration of stakeholders' input, methodology for an unbiased measurement of graduates' long-term attainment rate, as well as analysis and identification of a set of strategies for continuous quality improvement (CQI). These PEOs are mapped using the guidelines set by the Engineering Council to those required by the Engineering Accreditation Council (EAC), Malaysia. The outcome of the mapping exercise was used to formulate an anonymous online questionnaire survey as a measure of the PEOs' attainment. Key outcomes from this study revealed that graduates are broadly satisfied with their achievement in all eight PEOs. Strategies were also proposed to improve the attainment level in four PEOs with relatively lower attainment rate, as part of the CQI process adopted in the department. Keywords: accreditation, continuous quality improvement, outcome-based education, programme educational objectives Introduction Outcome-based Education (OBE) has become a central feature in the accreditation of engineering degrees offered by institutions of higher learning worldwide. For example, the assessment and evaluation of students' learning outcomes are the general criteria set by the Accreditation Board for Engineering and Technology (ABET) (ABET 2012) in the United States, as well as the Australian Graduate Attributes set by the Engineers Australia (2005). In the case of institutions of higher learning in Malaysia, the accreditation guidelines are set The Higher Education Academy 74 doi: /ened

2 Tshai et al. by the Engineering Accreditation Council (EAC) within the Board of Engineers Malaysia (BEM), which is also moving towards the adoption of OBE, as pointed out by the study of Aziz et al. (2005).The main reason for this high level of worldwide adoption is due to the fact that OBE advocates a paradigm shift from traditional practises of course credit accumulation to a focus on the students' achievement in high order learning and the mastery of cognitive thinking skills. This is believed to be a better measurement of the students' success in achieving the key necessary skills. Within an institution, OBE implementation requires the restructuring of the educational programme, courses, curriculum, assessment and reporting systems (Malan 2000). In the 1950s, the work of Bloom (1956) and his colleagues in developing taxonomies for educational objectives became important. The taxonomies eventually became benchmarks which were then used in the formulation of specific objectives and the development of targeted criteria to establish the learners' attainment of acceptable standards against the desired learning outcomes. Bloom's work, particularly in addressing the cognitive domain, remains instrumental in the assessment of OBE. In his work, Killen (2000) stressed the importance of linking assessments to long-term and significant outcomes expected of the learners, or to the short-term enabling outcomes which were derived from these long-term outcomes. This is in accordance to the principles of clarity of focus and designing back. There are two major types of outcomes in OBE, as defined by Killen (2000). The first emphasises on measurement of coursework, examination results, rates of course completion and employment upon graduation, while the second performance indicator is less tangible, commonly required the leaners to express what they have learned and capable to perform as a result of completing their education. It places greater focus on long-term outcomes leading to the future career success of the learner. OBE consists of four main principles (Brandt 1992): clarity of focus, expanded opportunity, high expectations and design down. Clarity of focus requires gearing the development of curriculum towards the outcomes expected of the students. These should not be short-term targets (weekly, semester or yearly) but end results that students would achieve upon graduation and beyond. This also emphasises on the need to pursue continuous development throughout the career. The second principle, expanded opportunity, aims to provide learners with various ways to achieve the learning outcomes. Approaches such as lectures, tutorials, assessments, seminars and workshops, industrial visits and dialogues, industrial attachment, departmental briefing, student sharing sessions and competitions are adopted within the OBE framework. The high expectations principle requires every learner to consistently achieve high level of performance. Students must develop a structural mind set and a sharp mind to tackle problems. They have to utilise the engineering principles and common sense efficiently to track sequential thoughts. Other attributes such as fast learning by organising information rather than just receiving information, holistic approach in handling engineering problems with appropriate safety, sustainable and engineering awareness are highly desirable. The final principle, design down, requires all curriculum design and teaching activities to comply with the outcomes from top to bottom levels. Spady (1994) illustrates the development of outcomes following the hierarchy of Exit Outcomes, Programme Outcomes, Course Outcomes, Unit Outcomes and Lesson Outcomes. The assessment of the overall achievement includes a linkage to long-term educational outcome of the learner, e.g. the long-term Programme Educational Objective (PEO) must be measured for a period of at least five years from graduation, as illustrated in Figure 1. Another key element in OBE focuses on the quality assurance of the education system. Quality measurement is important in engineering education (Owlia & Aspinwall 1996, 1998) The Higher Education Academy 75 doi: /ened

3 to ensure that the objectives of any engineering course are achieved and to identify key areas for improvement. There is a need to establish a continuous quality improvement (CQI) framework for engineering education to address the problem of skillset mismatch between the industries and institutions, as reported in recent surveys (Waks & Frank 2002, Magee 2004, May & Strong 2006). Figure 1 The proposed linkage of Spady s (1994) top to bottom levels of outcomes development with the incorporation of long-term PEOs. CQI is a process where an education programme is continuously, intentionally and systematically improved in a cycle or process where an increasing positive outcome is envisaged. CQI is relevant in an engineering degree programme at university because most accreditation engineering bodies have instituted CQI in the OBE. Several universities have adopted CQI in the ongoing assessment of their degree programme and have developed methodology to measure outcomes and to improve the teaching and learning process (Rozeha et al. 2007). Many have recorded marked improvements in the teaching and learning process over the period of implementation, such as those reported by Anuar et al. (2009) in their study at University Tenaga National, Malaysia and Shekar et al. (2008) at Purdue University Calumet, Hammond, IN. Input from stakeholders is an important component in the CQI process. During the three to four years of an engineering degree programme, various stakeholders and factors play significant roles in providing the most appropriate learning environment, innovative teaching style, assessment and feedback upon which a student's competency on his/her pathway to a professional engineering career is cultivated. Todd and Magleby (2005) reported a case study of developing a design capstone course with inputs from various stakeholders including students, faculty and administrators. The Department of Mechanical, Materials and Manufacturing Engineering at the University of Nottingham Malaysia Campus (UNMC) adopts a similar hierarchy which links the outcomes from the top to the bottom levels in the vision and mission of the university, PEOs, Programme Objectives (POs) and module Learning Outcomes (LOs). The attainment level of the exit outcomes are measured through an anonymous online questionnaire survey and the detailed analyses have been reported by Ho et al. (2013). Determining students' satisfaction level in their attainment of PEO in the survey is an essential part in closing the loop of the CQI process within the context of OBE. As an example, Sani et al. (2008) pointed out that using surveys in this enquiry is an essential tool for the CQI in their degree programme. This paper hence focuses on measuring the attainment level of the long-term PEOs for MEng graduates from the Department of Mechanical, Materials and Manufacturing The Higher Education Academy 76 doi: /ened

4 Tshai et al. Engineering at the UNMC (as indicated at the top level in the hierarchy shown in Figure 1). The next section provides an overview of the OBE and the hierarchy of the measurable outcomes, as well as the incorporation of CQI in the PEO formulation framework. The third section illustrates the methodology used in the assessment of students' attainment of PEOs. Results are discussed in the fourth section. The fifth section concludes this study and makes appropriate recommendations for future works. PEOs formulation framework At the UNMC, the stakeholders' involvement is an essential part of the review process. The opinions and requirements of both internal and external audits on quality assurance, i.e. both the university and external quality assurance bodies as stakeholders are regarded as crucial elements in the quality of the higher education provided. The PEOs are formulated in accordance to the procedure depicted in Figure 2. Figure 2 Procedure in the formulation of PEOs. The Higher Education Academy 77 doi: /ened

5 The PEOs should contain broad statements describing the career and professional accomplishments that the programme is preparing the graduates to achieve. Therefore it concerns various parties and the first step is to identify the stakeholders, as outlined as: the university, professional societies and regulatory bodies, members of staff, current student, external examiners, alumni (use for validation), industrial advisory board, employers (use for validation). The procedure outlined in Figure 2 enables the formulation of a set of eight PEOs capable of matching the major requirements of stakeholders, which was subsequently linked to the UK-SPEC (Standard for Professional Engineering Competence) Learning Outcomes (Engineering Council, UK 2010), as shown in Table 1. This set of learning outcomes is based upon the MEng learning outcomes set by the Engineering Council which comply with the UK-SPEC statements adapted by the Institution of Mechanical Engineers (IMechE) for Mechanical Engineering degrees. Here, they are categorised into various symbolic coding as in Table 1. Learning outcomes with m suffix are masters level enhancements to the specific learning outcomes covered in BEng programmes for satisfying the Chartered Engineer (CEng) academic requirements. Learning outcomes with M coding are additionally set by the department to addresses transferable/key skills which is shown in Table 2. In compliance with the university's Quality Manual, the Mechanical Engineering programme undergoes two formal review processes. Figure 3 shows the flow chart of the review processes leading to the achievement of the CQI objectives. The schools review is a major review and happens on a five year cycle and the annual programme review happens every year. These reviews encompass a wide range of feedback as well as several sub-processes starting from the frontline feedbacks from students via the Learning Community Forum (LCF), Students' Evaluation of Teaching (SET), Students' Evaluation of Module (SEM), students' performance in coursework and exam as well as evaluation of students' attainment of POs. These frontline feedbacks would be reviewed at UNMC and then jointly with counterparts from the UK campus. This can be used to formulate a new set of module outlines for the next teaching cycle if needed. The module reviews then formed a set of intermediate feedbacks for subject theme review conducted both locally at UNMC as well as jointly with the UK campus. The school review before the university's teaching and learning board is conducted using outcomes from UNMC and University of Nottingham UK programme review alongside stakeholders' feedback. Assessment methodology The main methodology used in measuring the attainment level of PEOs is through an online anonymous questionnaire survey. Fifty-one MEng graduates from the Department of Mechanical, Materials and Manufacturing Engineering at UNMC are selected as the target population for this study, with a total of 24 respondents. The attainment level of the PEOs is then analysed through the survey results. A dedicated set of survey questions must be customised for individual courses. The set of questions were strengthened with a focus group discussion consisting of members of staff in the department to formulate the survey. The Higher Education Academy 78 doi: /ened

6 Tshai et al. The outcomes of the focus group discussion were translated into an anonymous online questionnaire survey as a tool to measure the attainment of PEOs against a five discrete level gauge (from very satisfied to very dissatisfied). Table 1 The PEOs of UNMC's Mechanical Engineering undergraduate degrees mapped to the UK-SPEC Learning Outcomes. PEOs PEO1 PEO2 PEO3 PEO4 PEO5 PEO6 PEO7 PEO8 PEOs statement An understanding of the principles and techniques of mechanical engineering science and of the computational and mathematical tools used in the solution of real problems The ability to apply these principles and methods in the practice of mechanical engineering The ability to visualise, design and analyse components and assemblies in three dimensions A sound knowledge of material properties and manufacturing methods and their cost effective use in the design and improvement of engineering components and equipment The ability to communicate ideas effectively in written reports, verbally and by means of presentations to groups The ability to work as a member of a team and to display initiative and enterprise in the planning and realisation of engineering projects Confidence through practice in the use of experimental methods The appreciation of the responsibilities of Professional Mechanical Engineers and of the social, management and financial environment in which they work UK-SPEC statements adapted by the IMechE US1, US2, US3, US1m, US2m, M5, M6 US3m, US4m, E1, E2, E3, E4, E1m, P3, P4, US2 D1, D2, D3, D4, D5, D6, M5, D4m D1m, D4m, P1, P2, P7, P1m, P2m M1, M2, M3 M4, M7, US3m E2m, E3m S1, S2, S3, S4, S5, S1m, S2m, P5, P6, P8, P8m, M7, US4m, D3, D6, P1m, P7 Table 2 Learning Outcomes set by the department to address transferable skills. M1 M2 M3 M4 M5 M6 M7 Transferable/key skills (M) Ability to prepare technical reports Ability to give technical presentations Ability to communicate effectively Leadership skills and ability to as a team Ability to use general IT tools Develop a scientific approach in solving problems Ability to learn independently and to adapt a critical approach in investigation The online questionnaire survey contains 34 questions from six groups consisting of details, professional development, salary scale, job prospects, continuous professional training and the PEOs, as shown in Table 3. The Higher Education Academy 79 doi: /ened

7 Figure 3 Key review processes to facilitate CQI. All alumni graduated in the last five years were invited to respond to the survey. The questionnaire survey is designed to monitor the student's progress in their career since graduation. The survey is relatively short, simple and straightforward to encourage a higher level of participation. The purpose of the survey is also self-explanatory and made available to the alumni at the onset of the session, emphasising on the anonymity of the survey. Alumni who agreed to participate are given a web link to access the survey and the questionnaire is made available in a predefined period of time. They are given the option to withdraw at any time for the case where they do not feel comfortable to submit their survey. Upon obtaining the results, statistical analyses are conducted on the measured data. Quantitative data are analysed and depicted in graphs and charts to determine the The Higher Education Academy 80 doi: /ened

8 Tshai et al. percentage of a particular finding, with percentage of various elements of the PEOs being rated as high (above 70%), medium (50 69%) and low (49% and below) as a guideline to identify the most crucial elements of the PEOs for continuous quality improvement in the education system of the engineering programme. Table 3 The six major groups in online questionnaire survey. Group Categories 1. Details Student's degree Student's programme Year of graduation/study 2. Professional development Board of Engineers Malaysia (BEM) Institution of Engineers Malaysia (IEM) Institution of Mechanical Engineers, UK (IMechE) Other relevant professional institutions Class of membership 3. Salary scale Monthly salary scale 4. Job prospect Area/industry Job title Number of staff under your leadership 5. Continuous professional training Job training, self-initiated professional training Attendance of short course, seminar, workshop or conference 6. Programme Educational Objectives PEO1 to PEO8 (PEOs) Results and discussions Figure 4 shows the various industrial clusters in which the graduates who participated in the survey are employed. Three largest clusters are manufacturing, consultancy and oil and gas industry, which accounts for 50% of the total respondents. Figure 5 shows the outcomes of the questionnaire survey on the attainment of PEOs amongst alumni. From Figure 5, it can be observed that four PEOs (namely PEO1, PEO2, PEO5 and PEO6) achieved favourable level of attainment (>70%), whilst the remaining PEOs have achieved attainment levels lower than 70%. The PEOs under 70% (PEO3, PEO4, PEO7 and PEO8) would be addressed in the CQI process. Figure 4 Industrial clusters of the respondents employed in their career. The Higher Education Academy 81 doi: /ened

9 Figure 5 Results from questionnaire survey showing PEOs attainment amongst alumni. PEO1, PEO2, PEO5 and PEO6 achieved attainment rate above 70%. These PEOs are related to core principles of mechanical engineering and its application in practical problems, the ability to communicate ideas effectively using written formats and verbal presentation, working together as a team member displaying initiative and enterprise in engineering projects. These PEOs are the basis of the core structure of the mechanical engineering programme hence its coverage throughout the programme is eminent especially in the compulsory elements of the programme e.g. Professional Studies, Solid Mechanics, Materials and Manufacturing, Structural Vibration, Thermofluids, etc. Therefore, students had vast opportunities to work in a team in several modules featuring group work e.g. Design and Manufacture, Group Design Project and Group Design and Make. A 12 week industrial internship is compulsory for students, giving them the required exposure in applying their knowledge in a practical environment. Students are also strongly encouraged to participate in team competitions such as Bosch Power Drill Race, Formula Varsity, the James Dyson Award, etc. to cultivate the culture of team working and to build up the experiences in applying these principles in their career upon graduation. This has contributed to the graduates' confidence in the attainment of these PEOs. PEO3 has the lowest attainment rate of 54%. This is concerned with the graduates' ability to visualise, design and analyse components and assemblies in three dimensions. The achievement of this PEO is mainly relied on the delivery, contents and assessment of modules with design elements. The low attainment indicates that more emphasis should be placed in guiding the students in constructing three-dimensional (3D) engineering models in orthographic and auxiliary projections, isometric views, sectioning, dimensioning and assembly drawings, as well as their perceptions in conceptual, embodiment and detailed design. Strategies to improve the attainment of PEO3 formed part of a portfolio of CQI processes to be implemented in the programme. These include introduction of a comprehensive self-help guide on Moodle (the university's Virtual Learning Environment), employing more research assistants with knowledge of 3D modelling, engaging industrial lecturers to contribute to the teaching of design, continuously updating and improving the video demonstrations used, improving the efficiency of the assessment of these elements in the programme. PEO4 is mostly linked to the graduates' understanding on materials properties and manufacturing, as well as cost effective in design and system improvement. This is related to the design and practice in the UK-SPEC Learning Outcomes. Only 63% of the graduates felt that they have acquired this educational objective and applied it in their career. A relatively low attainment rate indicates that the graduates need more practicing The Higher Education Academy 82 doi: /ened

10 Tshai et al. elements in costing as well as creative and innovative approach to improve an engineering system and perhaps this has to be implemented in the curriculum design. Another possible reason is that the graduates may not have the opportunity to apply the skill in their career due to various reasons such as the job scope and the nature of the business in which the employer is involved in. Suggested improvements for better attainment in this PEO include embedding more practical elements in the design, materials and manufacturing modules. For example, students can be asked to submit a business plan which includes costing, business plan as well as the marketing strategy in their design project. Optimisation in terms of material design, which seeks a balance between low cost materials and system performance, would also need to be emphasised. The lower attainment level of PEO7 highlighted that the graduates do not have sufficient confidence to practice and effectively utilise systematic experimental methodologies in engineering problem-solving processes. Lack of confidence in experimental methods could be correlated to the students' concern on the health and safety risk associated with a particular experimental procedure, and their ability to extract data pertinent to an unfamiliar problem, apply mathematical/computer-based models to formulate feasible solutions, assess the limitations of particular cases, setup and conduct appropriate experimental methods to evaluate the solution. The challenge to improve the attainment level in PEO7 dictates a necessity to empowering students with a skillset of experimental techniques, as well as an awareness of the associated risk. This unavoidably requires a restructuring of modules delivery with laboratory experimentation and machining workshop. Apart from a set of general health and safety rules and regulations, the associated risks must be customised and adapted to suit a dedicated laboratory/workshop. In addition to the underpinning engineering knowledge, experimental methods typically require a hands-on approach, whereby a combination of what the students see, hear, and touch provide value-added activities to boost the confidence level. In this respect, small group teaching should be further enhanced in laboratory activities to provide each student an opportunity to conduct hands-on experiments. Often, the assessment on student comprehension in experimental procedure has not been rigorously tested as the focus is largely on the measured data and their subsequent engineering calculations. This element should be improved by considering the various level of Bloom's taxonomy in setting the laboratory evaluation sheet and having the instructors to pay closer attention to ensuring that proper best practises are followed. PEO8 is on the expectation of the graduates to appreciate the responsibilities of Professional Mechanical Engineers and of the social, management and financial environment in which they work. The results show that only a small proportion of the graduates believe that their degree has provided sufficient coverage and exposure to the requirements, expectations and standards regarding the responsibilities of a professional engineer, as well as little emphasis on elements in management and business and financial knowledge so that an engineer could carry out his/her work efficiently. For better attainment in PEO8, embedding elements of professional practices of an engineer into parts of the classical modules is proposed. Although there is a module on Professional Studies, this is insufficient as the module only carries 10 credits (8.3% of Year One modules). When taught in isolation, it has become difficult for the students to relate professional practices to engineering knowledge therefore embedding these elements into core modules is preferable. Conclusions Using a pre-determined set of PEOs complying with the UK-SPECS outcomes, this paper reports a survey to measure the graduates' attainment level of PEOs from their degree The Higher Education Academy 83 doi: /ened

11 course within five years of graduation from the UNMC Mechanical Engineering programme. The survey results have shown that graduates have perceived their attainment of most PEOs to be above the satisfactory level. The results also showed that there are four areas (PEO3, PEO4, PEO7 and PEO8) with relatively lower attainment as rated amongst the graduates. Using these results, insights into how the graduates perceived their own attainment and key causes of unsuccessful attainment were highlighted. Appropriate recommendations were proposed using the UK-SPECS and BEM guidelines. This process is part of the departmental CQI process strategy and changes are to be implemented in the next academic year to improve the level of attainment of these PEOs. The conclusions of the study as discussed is limited to the results from the graduate questionnaires on the attainment of the programme of study and do not consider other factors beyond graduation which may affect the results of the survey. There are several issues to be addressed in further works from this study which include assessing the graduates' individual exposures/experiences and further professional development schemes beyond graduation. Inclusion of feedbacks from employers should also be considered to make the study more comprehensive. References Accreditation Board for Engineering and Technology (ABET) (2012) Criteria for accrediting engineering programs Baltimore, MD: ABET. Anuar, A., Shuaib, N., Sahari, K. and Abidin, I. (2009) Continual improvement and assessment plan for mechanical engineering programme at UNITEN, pp Kuala Lumpur: IEEE. Aziz, A., Megat Mohd Noor, M., Abang Ali, A. and Jaafar, M. (2005) A Malaysian outcome-based engineering education model. International Journal of Engineering and Technology 2 (1), Bloom, B.S. (1956) Taxanomy of educational objective: handbook 1. Cognitive domain. New York: McKay. Brandt, R. (1992) On outcome-based education: a conversation with Bill Spady. Educational Leadership 50 (4), Engineering Council, UK (2010) The accreditation of higher education programmes: UK standard for professional engineering competence. London: Engineering Council, UK. Engineers Australia Accreditation management system educational programs at the level of professional engineer. Barton, ACT: Engineers Australia. Ho, J.-H., Tshai, K.Y. and Yap, E.H. (2013) Students' Attainment of Programme Objectives (POs) for the Mechanical Engineering Undergraduate Degrees at the University of Nottingham Malaysia Campus (UNMC), International Symposium on Education, Psychology and Social Sciences (ISEPSS 2013). Kuala Lumpur, Malaysia. Killen, R. (2000) Standards-referenced assessment: linking outcomes, assessment and reporting. Keynote address at the Annual Conference of the Association for the Study of Evaluation in Education in Southern Africa, Port Elizabeth, South Africa, September. Magee, C.L. (2004) Needs and possibilities for engineering education: one industrial/ academic perspective. International Journal of Engineering Education 20 (3), Malan, S.P. (2000) The "New Paradigm" of outcome based education in perspective. Journal of Family, Ecology and Consumer Sciences 28, The Higher Education Academy 84 doi: /ened

12 Tshai et al. May, E. and Strong, D. (2006) Is engineering education delivering what industry requires. The Third CDEN Design Conference, Toronto, Ontario, Canada. Owlia, M. and Aspinwall, E. (1996) A framework for the dimension of quality in higher education. Quality Assurance in Education 4 (2), Owlia, M. and Aspinwall, E. (1998) A framework for measuring quality in engineering education. Total Quality Management 9 (6), Rozeha, A.R., Mazlina, E., Sadruddin, M. and Abdullah, R. (2007) Continuous improvement in engineering education through effective value for money audits. In Proceeding, Second International Conference on Engineering Education & Training (ICEET-2), MOVENPICK Hotel Kuwait, Kuwait City, Kuwait. Sani, M. et al. (2008) Assesment of the mechanical engineering programs by exit surveys at University Malaysia Pahang, pp Shah Alam, s.n. Shekar, C.R., Farook, O. and Bouktache, E. (2008) Continuous improvement process based on outcome based education. Nashville, TN: IAJC-IJME. Spady, W. (1994) Choosing outcomes of significance. Educational Leadership 51 (6), Todd, R.H. and Magleby, S.P. (2005) Elements of a successful capstone course considering the needs of stakeholders. European Journal of Engineering Education 30 (2), Waks, S. and Frank, M. (2002) Engineering curriculum versus industry needs: a case study. IEEE Transactions on Education 43 (3), The Higher Education Academy 85 doi: /ened