Quality Assurance in a PBL Learning System in Engineering and Science Education

Transcription

1 Quality Assurance in a PBL Learning System in Engineering and Science Education Palle Qvist 1, Egon Moesby 2 1 Department of Development and Planning, Aalborg University, DK-9220 Aalborg Ø, Denmark (palle@plan.aau.dk) 2 Head of Studies for School of Basic Studies, Aalborg University, DK-6700 Esbjerg, Denmark (hem@aaue.dk) Abstract Evaluation is a central element in the overall quality assurance of teaching and learning in higher education. For a Study Board or study unit is it important to have designed assessments procedures directed towards management, teaching and learning. Among theorists many epistemological and ideological positions exists on the nature of evaluation, on how to conduct one as well as on how to present and on how to use the generated results [1]. The School of Basic Studies at Aalborg University, Denmark, operates an internal quality assurance system which combines summative and formative assessment methods at different levels. Internal evaluation is according to Love [2] characterized by the use of internal staff or contractors closely bound to the organization to conduct evaluation activities. Programs and problems of direct relevance to the management of the organisations is the usual focus. Remedying problems, not only diagnosing them and developing recommendations, are a purpose. The faster problems can be remedied the better, for the benefits of the organisation, the clients (the students and professors) and the stakeholders, internal (study boards of later semesters) and external (industry). The article describes the setup of the internal evaluation system in detail. Keywords: Internal quality assurance, PBL, POPBL, program evaluation, Aalborg Model 1. INTRODUCTION 1.1 Definitions and use of terms The School of Basic Studies of Engineering, Science and Medicine at Aalborg University, Denmark, operate a unique and comprehensive internal quality assurance system which combines summative and formative evaluation forms and methods at different levels. The term evaluation is in this article understood as the valuing of a process, a product or an event [3]. The valuing happens by use of different methods with the outcomes of processes, products or events. The outcomes are to judge the worth of measures, values, descriptions, norms or criteria s. Is the purpose with the valuing progressive to identify possibilities and potentials for change or improving then the evaluation is formative. Is the purpose retrospective and judging in relation to processes, products or events which have occurred then the evaluation is summative.

2 Formative evaluation may be built on summative elements; however the summative results are used for progressive changes. evaluations can be used progressively. If somebody want to change something which is, then it is necessary to know how things are, to know the status. Form Purpose Formative evaluation evaluation Progressive aimed to identify possibilities and potentials for changes and improvements Retrospective and judging. Establish status. TABLE 1. The two main evaluation categories The term cohort or career group is used for a number of students (up to 100) within the same career. The term team or group is used for one team of 6-7 students. Each team is formed under a specific career, see Figure 1. FIGURE 1. Group structure at The School of Basic Studies

3 The Quality Assessment Group (QAG) which operates part of the evaluation program consists of 3 teachers nominated by the Head of Studies and working under a declaration decided by the Study Board. The evaluations at the School of Basic Studies are mainly out-come based. Schalock [4] defines outcome evaluation as a process that lead to judgements and decisions about programs and policies. A program is understood as a set of operations, actions or activities. Those activities, actions or operations are designed to generate specific formulated outcomes. His definition of outcome-evaluation sounds: "A type of evaluation that uses person and organization-referenced outcomes to determine current and desired person - and program-referenced outcomes and their use (program evaluation), the extent to which a program meets its goals and objectives (effectiveness evaluation), whether a program made a difference compared to either no program or an alternative program (impact evaluation) or the equity, efficiency or effectiveness of policy outcomes (policy evaluation) [5] Data used in the evaluation program at the School of Basic Studies are mainly obtained by the use of online questionnaires with open and closed answer possibilities, fully structured and with predetermined categories [6][7][8] and focus group dialogue. The dialogue in the focus groups are structured with the help of an agenda but everybody can address any subject. A focus group dialogue can often contribute with nuanced and deeper viewpoints than other methods of data collection although it might be difficult to control the dialogue because of different opinions and perhaps conflict of interests [9][10]. Table two shows the types of data, how they are obtained, and the main focus of the evaluations and frequency of evaluation of the general program at the Study Board. Data type Obtained by Focus Frequency Quantitative/ Qualitative Questionnaire to all students Learning, teaching and administrative outcomes Bi-annual Formative Qualitative Minutes from chairmen s group, vice chairmen s group, steering groups Learning, teaching and administrative outcomes Improving and change of program Bi-annual Quantitative Results of exams Monitoring grades Bi-annual TABLE 2. Overview of types of data obtained in the process of assuring quality at The School of Basic Studies. Table 3 show comparisons between two or more types of data obtained and analysis techniques. But also different kinds of inquiry designs.

4 Form Methods/Inquiry Designs Operator and operating unit Monitoring grades Administration Frequency Biannual The Secretariat Online questionnaire QAG Bi-annual Formative Focus group dialogue Group of career coordinators Bi-annual Focus group dialogue Group of vice-chairs Bi-annual Formative Focus group dialogue Steering committee 4-6 times per year Formative TABEL 3. Methods and units operating the quality assurance system at The School of Basic Studies. The roots of mixed-methods inquiry in science is found in the construction of triangulation. Triangulated assessing involves the use of multiple methods - each representing an alternative perspective. The purpose is to improve the confidence and validity of the evidence. If, for instance, the data obtained from qualitative questionnaires correspond the data obtained from focus group interviews, then the results are believed to be more valid than if entirely based of just one of the two findings [11][12][13]. As pointed out by Webb and others: "Once a proposition has been confirmed by two or more measurement processes, the uncertainty of its interpretation is greatly reduced. The most persuasive evidence comes through a triangulation of measurement processes. If a proposition can survive the onslaught of a series of imperfect measures confidence should be places in it" [14] [15]. When data is contradictory, it is important to be open and prepared for closer and critical analysis and obtaining of further data before reaching any conclusion. When combining quantitative and qualitative methods with each other, one also combines what is called realism and objectivity with value-judgemental contextual knowledge [16]. Mixing methods allows getting a more differentiated understanding of the development and level of quality in a program. It is important, however, to plan and practise triangulation techniques that are easy to use and thrust worthy [17] and take the criticism of mixing methods into consideration [18].

5 2. THE QUALITY ASSURANCE SYSTEM 2.1 Description The Quality Assurance System at The School of Basic Studies is developed over more than 10 years. Elements of the system is also found in the quality assurance policy of the Faculty of Engineering, Science and Medicine, to which The School of Basic Studies belong [19]. It is closely related to the structure of the organisation as shown in the chart below.. FIGURE 2. The organisational chart of The School of Basic Studies The Head of Studies and the Study Board are heading the organisation. Each career group of which there in 2007/08 are 20 at 3 campuses are headed by a chairman and vice chairman. A steering group of each cohort consist of one student from each project group (electing the vice chairman) and 2 lecturers/facilitators. The purpose of the quality assurance system is to: 1. Create a base which makes it possible for the Study Board to evaluate the study program and control to which degree the reached learning outcomes are in accordance with the expressed Expected Learning Outcomes (ELO s); 2. To motivate and support the teachers to act critical and reflective towards their teaching in relation to substance as well as to teaching forms; 3. To motivate and support the students to act critical and reflective towards their learning process and study efforts; 4. To motivate and support the students in taking independent responsibility of their learning and reflect on how best to improve their study efforts in their continued studies.

6 The quality program is described in the Study Regulation [20]. An important element in assuring quality is the continued development and improving of a handbook for the chairmen of cohorts in which the common rules and standards are described (Faggruppemanual). An overview of the units operating the quality assurance system at the School of Basic Studies is shown in table 3. Part of the program is operated by an independent unit named Quality Assessment Group (QAG) under the Head of Studies. The QAG consists of 3 teachers nominated by the Head of Studies and working under a declaration decided by the Study Board. According to the declaration QUG as an example make, distribute and adapt questionnaires to the students with the objective to obtain information about the general level of contentment of the study program and the teaching and learning, but also about their judgement on how the study program and the teaching contributes in reaching of the educational objectives (Authors translation [21]). The questionnaire is distributed bi-annual at the end of each semester. It is an online questionnaire and the students are asked to give their opinion of their general satisfaction with the Basic Studies, the degree to which the semester contributed to reaching the ELO s stated in the program in relation to their ability to work in a scientific way i.e. methods, theories and models, in relation to analysing and solving problems, planning and managing problem based project work, reflect and generalise from their learning throughout the semester and communicate and publish the results of the project work in a accordance to scientific traditions. They are also asked to which degree they see their facilitators, subjects, administration, IT support unit having contributed to their reaching of the ELO s. Due to the fact that the evaluation is carried out online and the participation is voluntarily, the number of responders usually is low. It means that the obtained data cannot in its own - be used as argument for decision for action by the Study Board [22][23]. If other data obtained through other methods are equal or points in the same direction, the Study Board may consider taking action. The semester s summative evaluation of the entire program conducted by QAG gives the student respondents the possibility on an individual level to give their opinion about the study program. The opinion between the students might change the next semester or the next year. However, if unacceptable tendencies are recognised over the years then it will call for change. Because the evaluation is performed as a summative evaluation, any cause for change can only be initiated after the semester, which underline a lack of real time evaluation, in which immediate can be taken. QAG publish the results in a report for the Study Board and published on the internet. The report contains remarks if QAG thinks that the obtaining of the outcomes is critical. The opinion of QAG is in the Study Board balanced with evaluations on other levels, e.g. minutes from the steering groups. Each career group form a formal steering committee. The steering committee consists of one representative from each of the team within the career group. In addition, three members of the VIP staff are represented, and the chair (coordinator) of each career group acts as chair of the steering committee. The Vice-chair is elected amongst the student representatives. The steering committee acts as a target group in relation to real time evaluation and quality assurance. The committee have monthly meetings, and the summative and formative real time evaluation serve to be able to immediate action if needed and to prevent malfunctions. The meetings follow a standard agenda designed by the Study Board to ensure coverage of the desired relevant topics for the quality assurance tasks. If a topic not listed on the standard agenda is raised, this can be added to the agenda and discussed in the committee. Minutes are taken by the Vice-chair, and sent to the Head of Studies and is later published on the internet. The following main topics are included in the standard agenda; however, the template is more detailed in order to make it operational. 1. Evaluation of facilitation and learning in the groups; 2. Evaluation of the courses and lecturing; 3. Evaluation of Administration and related services; 4. Evaluation of IT services..the idea is that the members of the steering committee should discuss the agenda prior to the meeting with the team members to find a common viewpoint. However, team members have often diverse opinions and some might be omitted. Sometimes the representative must trust own opinions or feelings of what is the opinion of the group.

7 The Study Board discuss and evaluate the minutes. As a part of its duty, QAG has to overview/monitor the minutes and report on critical conditions. A steering group can also decide to let the Chair bring forward matters related to administration, teaching and quality assurance by for instance bringing a case to the knowledge of the Study Board, the Head of Studies or teachers involved. The career coordinators form a special coordinator group, which as well is defined as a focus group. This group has monthly meetings with the Head of Studies, and during these meetings, relevant topics regarding teaching and learning, and topics related to the logistics, administration, IT support, and service in general are discussed. The coordinators are appointed by the career Study Heads on B.Sc. and M.Sc. level, as illustrated in Figure X. This way the vertical coordination for the entire career planning is coherent. The coordinator group serves as well as a very useful discussion partner for the Study Head, as politics and strategies can be discussed on the basis of a practical perspective prior to Study Board meetings, as the coordinators represents teachers with experience related to teaching and educational policy making. The group of vice-chairs of the career groups form an independent body which acts as focus group in relation to quality. The Head of Studies arrange normally two meetings each semester with this group of students. All questions in relation to quality, the study, the teaching, administration and services offered by The School of Basic Studies can be brought to the agenda. Even though topics are expected to have been discussed in the steering committees, the representatives sometimes express what they think is the opinion of their study mates or what they think themselves because the topic has not been discussed between the students they represent. The quality program further involves monitoring of marks given at the exams. Students are assessed in relation to the outcomes of subjects and project work. The assessments are summative. At each examination an external examinator participates. The examinator and external examinator give each a grade and the average is the final grade given. The external examinator can be VIP staff from other departments within the university or appointed from industry or from other universities. At the second semester of The School of Basic studies it is always external censor from the industry or other universities who participates at the project examination. During the first year of study more than 100 different professors or representatives from industry acts as external examinator at the School of Basic Studies. By monitoring the marks the Study Board can overview the performance of the students judged on impartial judgements from the independent external examinators. If the average marks vary too much from the expected standards, the Study Board has to look into reasons for the deviation, and take measures to correct the situation. 3. CONCLUSION The School of Basic Studies at Aalborg University, Denmark, operates an internal quality assurance system which combines summative and formative assessment forms and methods at different levels. Internal evaluation is according to Love [24] characterized by the use of internal staff or contractors closely bound to the organisation to conduct evaluation activities which also is the case at The School of Basic Studies. While summative assessments are retrospective and judging, formative methods are progressive and help to identify possibilities and potentials for change [25][26]. The two forms in combination give the management the possibility to identify different types of problems. Using the forms at different levels at the overall program level, career groups, group and individual level The Head of Studies and the Study Board are given the possibility to identify the need for proper action or change. Programs and problems of direct relevance to the management of the organisations is the focus. Remedying problems, not only diagnosing them and developing recommendations, are a purpose. The faster problems can be remedied the better, for the benefits of the organisation, the clients (the students and professors) and the stakeholders (internal and external). According to Stufflebeams classification of 22 evaluation approaches, the system developed is a management information system [27]. It is developed over the past 10 years and elements of the system are also found in the quality assurance policy of the Faculty of Engineering, Science and Medicine, to which The School of Basic Studies belong [28].

8 A management informative system supply the manager with information needed to conduct and report on the program in which he or she is in charge. It provides information that managers need to implement and report on management responsibilities. The approach has similarities with decisions/accountability-oriented approaches which provide data needed to development as well as to defending a program. A management information system has e.g. focus on objectives, specified activities and events. It gives information which needed to plan, direct, control, report and develop the program. Addressed by questions are between others people running the system or people using the system. Since Deming [29] stressed that managers should play close attention to processes rather than to outcomes, focus has also been on processes. Deming s argued that close attention to processes would result in better outcomes. The decision/accountability-oriented approach is also found in model at the School of Basic Studies, although not in the full scale as outlined by Stufflebeam. The approach is recognised by its purpose to judge - retrospective and in real-time - the value merits the worth of a program and proactively to help improve the program. 4. EPILOGUE 4. 1Matching the standards 45 European Ministers of Education adopted in 2005 the "Standards and Guidelines for Quality Assurance in the European Higher Education Area"(EHEA) as part of the Bologna process [30]. Although the standard and guidelines are for institutions - and The School of Basic Studies is not an institution but a program within Aalborg University - the assurance quality system of The School described above matches the internal quality assurance standards and guidelines of EHEA [31]. When it comes to a policy and procedures for quality assurance the School of Basic Studies has developed and implemented a strategy for the continuous enhancement of quality". It has a formal status and is publicly available as required in the standards and guidelines. The students play an important role at more levels as do the most important stakeholders - the Study Boards, which recruits students from the School of Basic Studies - through their representative: the chairmen of the career groups. According to the guidelines "confidence of students and other stakeholders... is more likely to be established and maintained through effective quality assurance activities which ensure that programmes are well-designed, regularly monitored and periodically reviewed". The quality assurance program described above monitor the progress and achievements of students and ensures their participation in the quality assurance activities. Also when it comes to collecting, analysing and use of relevant information for effective management of the programme of study and other activities are the requirements met quite well. Quoting the guidelines: "It is important that institutions have the means of collecting and analysing information about their own activities. Without this they will not know what is working well and what needs attention, or the results of innovatory practices. The quality-related information systems required by individual institutions will depend to some extent on local circumstances, but it is at least expected to cover: student progression and success rates; employability of graduates; students satisfaction with their programmes; effectiveness of teachers; profile of the student population; learning resources available and their costs; the institution s own key performance indicators" [32]. Employability, profile of student population and learning resources are not part of the quality assurance program described about. Out site the responsibility of The School of Basic Studies are quality assurance of teaching staff, because the staff are required from the Institutes of the University. But the standard and guidelines sound like this:

9 Institutions should have ways of satisfying themselves that staff involved with the teaching of students are qualified and competent to do so. They should be available to those undertaking external reviews, and commented upon in reports Teachers are the single most important learning resource available to most students. It is important that those who teach have a full knowledge and understanding of the subject they are teaching, have the necessary skills and experience to transmit their knowledge and understanding effectively to students in a range of teaching contexts, and can access feedback on their own performance. Institutions should ensure that their staff recruitment and appointment procedures include a means of making certain that all new staff have at least the minimum necessary level of competence. Teaching staff should be given opportunities to develop and extend their teaching capacity and should be encouraged to value their skills. Institutions should provide poor teachers with opportunities to improve their skills to an acceptable level and should have the means to remove them from their teaching duties if they continue to be demonstrably ineffective [33]. Missing in relation to the standards are participation of external reviewers and - perhaps irrelevant because it is a school of basic year studies - regular feedback from employers, labour market representatives and other relevant organisations. The standards and guidelines related to assessment of students are met through the Framework Provisions of the Faculty and the Study Regulation of the School of Basic Studies. References: [1] D.L. Stufflebeam, G. F. Madaus, & T. Kellaghan, (2000) Evaluation Models. Viewpoints on Educational and Human Services Evaluation. Second Edition. Hingham, MA, USA: Kluwer Academic Publishers. [2] A. W. Love (1983) Developing effective internal evaluation. New directions for program evaluation. San Francisco: Jossey-Bass [3] P. Qvist (2005) Formativ evaluering. IPN: Odense [Online] Available from: [Accessed 14 February 2008] [4] R. L. Schalock (2000) Outcome Based Evaluation Second Edition. Hingham, MA, USA: Kluwer Academic Publishers [5] R. L. Schalock (2000) Outcome Based Evaluation Second Edition. Hingham, MA, USA: Kluwer Academic Publishers [6] P. Hansen og P. Qvist (2006) Samfundslex. 3. udgave, 1. oplag. Gyldendal: København [7] C. Lewin (Ed.) (2004) Research Methods in the Social Sciences. Sage Publications, London [8] C. Robson (2007) How to do a Research Project. Blackwell Publishing: Oxford [9] I. Andersen (1999) Den skinbarlige virkelighed om valg af samfundsvidenskabelige metoder, 1. udgave, 3. oplag. Samfundslitteratur: Frederiksberg C [10] C. Lewin (Ed.) (2004) Research Methods in the Social Sciences. Sage Publications, London [11] J. C. Greene, H. Kreider & E. Mayer (2004) Combining Qualitative and Quantitative Methods in Social Inquiry. In Lewin, Cathy (Ed.) (2004) Research Methods in the Social Sciences. Sage Publications, London [12] N. K. Denzin (1989) The research Act. Third Edition. New Jersey:Prentice Hall [13] E. J. Webb, D. T. Campbell, R. D. Schwartz and L. Sechrest (1966) Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally [14] R. G. Owens, C. R. Steinhoff and S. M. Rosenbaum (1989) A Triangulation Methodology in Research on Social Cultures. Paper presented at the Annual Meeting of the American

10 Educational Research Association (San Francisco, CA, March 27-31, 1989). [Online] Available from: Value_0=ED306664&ERICExtSearch_SearchType_0=no&accno=ED [Accessed 20. February 2008] [15] E. G. Guba and Y. S. Lincoln (1981) Effective Evaluation: Improving the Usefulness of Evaluation Results Through Responsive and Naturalistic Approaches. Jossey-Bass: San Francisco [16] J. C. Greene, H. Kreider & E. Mayer (2004) Combining Qualitative and Quantitative Methods in Social Inquiry. In Lewin, Cathy (Ed.) (2004) Research Methods in the Social Sciences. Sage Publications, London [17] R. G. Owens, C. R. Steinhoff and S. M. Rosenbaum (1989) A Triangulation Methodology in Research on Social Cultures. Paper presented at the Annual Meeting of the American Educational Research Association (San Francisco, CA, March 27-31, 1989). [Online] Available from: Value_0=ED306664&ERICExtSearch_SearchType_0=no&accno=ED [Accessed 20. February 2008] [18] A. Massey (1999) Methodological Triangulation. Or How To Get Lost Without Being Found Out. In Massey, A. and G. Walford (Eds.) (1999) Explorations in methodology,studies in Educational Ethnography, Vol.2, JAI Press: Stamford [19] Framework Provisions (2007),Technical Sciences, Natural Sciences,Health Sciences,The Faculties of Engineering, Science and Medicine. Aalborg University: Aalborg [Online] Available from [Accessed 20 February 2008] [20] Studieordning (2007) Basisåret, Aalborg Universitet Aalborg, Esbjerg, København [Online] Available from [Accessed 20 February 2008] [21] Kvalitetssikring, Det Ingeniør, Natur- og Sundhedsvidenskabelige Basisår [Online] Available from: [Accessed 14 February 2008] [22] P. Qvist, I.-L. Hansen og B. Rønsholt (2006) Semesterevaluering P1; Det Ingeniør-, Natur- og Sundhedsvidenskabelige Basisår. Resultater af spørgeskemaundersøgelse. Aalborg. [23] P. Qvist, I.-L. Hansen og B. Rønsholt (2007). Semesterevaluerng P2; Version A. Det Ingeniør-, Natur- og Sundhedsvidenskabelige Basisår. Resultater af spørgeskemaundersøgelse. Aalborg. [24] A. W. Love (1983) Developing effective internal evaluation. New directions for program evaluation. San Francisco: Jossey-Bass [25] J. George and J. Cowan (1999) A Handbook of Techniques for Formative Evaluation. London: Kogan Page [26] P. Qvist (2005) Formativ evaluering. IPN: Odense [Online] Available from: [Accessed 14 February 2008] [27] D. L. Stufflebeam (2001) Evaluation Models. New Directions for Evaluation, no. 89, Spring 2001, Jossey-Bass: San Francisco [28] Framework Provisions (2007),Technical Sciences, Natural Sciences,Health Sciences,The Faculties of Engineering, Science and Medicine. Aalborg University: Aalborg [Online] Available from [Accessed 20 February 2008] [29] W. E. Deming (1986) Out of the Crisis Cambridge, MA: Center for Advanced Engineering Study, Massachusetts Institute of Technology [30] ENQA, European Association for Quality Assurance in Higher Education (2005) Bergen Communique [Online] [Accessed 20 February 2008]

11 [31] ENQA, European Association for Quality Assurance in Higher Education (2005) Standards and Guidelines for Quality Assurance in the European Higher Education Area [Online] Available from [Accessed 20 February 2008] [32] ENQA, European Association for Quality Assurance in Higher Education (2005) Standards and Guidelines for Quality Assurance in the European Higher Education Area [Online] Available from [Accessed 20 February 2008] p. 18 [33] ENQA, European Association for Quality Assurance in Higher Education (2005) Standards and Guidelines for Quality Assurance in the European Higher Education Area [Online] Available from [Accessed 20 February 2008] p. 17