Authors SEFI-paper Doing the right things right Quality enhancement in Higher Education

Transcription

1 Authors SEFI-paper Doing the right things right Quality enhancement in Higher Education Corresponding Author: Pernille Hammar Andersson Educational Consultant LearningLab DTU Study Programmes and Student Affairs Anker Engelunds Vej 1 Building 101 A DK-2800 Kgs Lyngby Denmark pea@llab.dtu.dk Other Authors: Peter Munkebo Hussmann Evaluation Consultant LearningLab DTU Study Programmes and Student Affairs Anker Engelunds Vej 1 Building 101 A DK-2800 Kgs Lyngby Denmark pmh@llab.dtu.dk Helge Elbrønd Jensen Dean of Education Study Programmes and Student Affairs Anker Engelunds Vej 1 Building 101 A DK-2800 Kgs Lyngby Denmark hej@adm.dtu.dk 1

2 Doing the right things right Quality enhancement in Higher Education. P. H. Andersson 1, P.M. Hussmann 1 and H.E. Jensen 2 1 LearningLab DTU, Technical University of Denmark, Lyngby, Denmark, pea@llab.dtu.dk, pmh@llab.dtu.dk 2 Dean of Education, Technical University of Denmark, Lyngby, Denmark, hej@adm.dtu.dk Abstract Are we doing the right things and are we doing them right? These questions should be subject to constant attention within Higher Education Institutions in pursuit of high quality educations and attracting students nationally as well as internationally. And due to the international development in Higher Education (HE), external quality assurance and improved accountability is becoming increasingly important. At the Technical University of Denmark (DTU), an internal quality enhancement system has been developed in order to evaluate and further develop the education programs. The overall objective is to ensure that the students receive education of excellent quality when they choose to study at DTU. Furthermore, the aim is to generate feedback on the implementation and apparent success of the three plus two degree structure adopted some years ago, in accordance with the objectives of the Bologna Process. A third aim is faculty and organisational development in order to enhance a deeper understanding of how to create quality in education programs. The system implies that two programs evaluate each other on the basis of five overall criteria. The criteria are descriptive and explain to teachers, directors of programs, and students the important factors that enhance quality in HE and strengthen the possibilities for the students to learn in a good and efficient way. So far, it seems that the evaluation process supplies the involved teachers and directors of study programs with relevant tools for making informed adjustments in their teaching and programs. Secondly, it contributes to establishing a learning community among the teachers with a growing common understanding of what is considered good quality in engineering education. This paper describes the internal quality enhancement system and analyses its impact on the quality of the teaching and learning at DTU. It describes how the internal evaluations prepare for external accreditations strengthening the latter by avoiding the risk of universities box-ticking their way to purchased credibility. Keywords: Quality assurance, teaching development, faculty development, quality enhancement 1. INTRODUCTION During the last three decades, quality assurance has been an integral part within European Higher Education (HE). Many different concepts have flourished in this growing quality movement, and with the Bologna process as the leitmotif, a common ground seems to be evolving. Thereby quality assurance has become a central activity at many universities at all organisational levels. Demanding high quality in all services and activities seems to be a general trend in all parts of society. However, it is not always made explicit what is actually meant by quality. Quality is somehow a very indistinct concept and can probably be placed in the category of concepts which can be identified as an essentially contested concept, very much depended on personal preferences as well as being multidimensional and complex 1. And since the quality of teaching and learning are some of the most complex and difficult areas to measure the increasing demand for and focus on quality in HE is even more challenging 2. 2

3 At the Technical University of Denmark (DTU) an internal quality enhancement system has been developed. The system attempts to deal with this complexity of quality in HE while developing the quality of the programs and teaching activities at the university by enhancing understanding of teaching and learning among faculty members. 2. QUALITY ASSURANCE IN HIGHER EDUCATION State control of HE has become more predominant 3, and the advent of market forces in HE has challenged the academic oligarchy 4. For quite some time accountability has been the buzzword in order to ensure value for public money 5. The growing quality movement in HE has resulted in a large diversity of quality assurance processes and actors in Europe, and even before the Bologna declaration made its impact, this development called for the next generation of quality assurance. 6 Initially these discussions were not very prominent in the Bologna process 7. In fact one of the main rationales for the initiatives in the Bologna declaration was to strengthen the international competitiveness of the European system of HE because Europe had lost its dominant position as a destination for overseas study 8. Thus, attracting students to European HE was a crucial task, and as Saarinen demonstrates 9 the meaning and understanding of quality in the Bologna process moved from promotion of competitive edge to establishing quality assurance techniques including program or institutional accreditation. The pressure towards more quality assurance has continued to increase, and QA has become an essential building brick in the European Higher education area 10. The main idea in the Bologna process concerning QA is that the primary responsibility for quality assurance in higher education lies with each institution itself 11. This implies, though, that national quality assurance systems should be established and include an accreditation system and evaluation of programs or institutions, including internal assessment, external review, participation of students, and the publication of results. In Denmark, a national accreditation agency for HE (ACE Denmark) was created in The agency will accredit all new and existing educations. ACE Denmark employs the well known method for assessing quality by using self evaluations, site visits, peer review, and external expert panels. Even though the procedure used by ACE Denmark and other Accreditation agencies seems reasonable and effective, there is a risk that universities strive to do the right things without focusing on doing them right. Due to the increasing demand for documenting results and outcomes, e.g. qua key performance indicators like student performance, student employment, faculty-student ratio etc, they risk focusing too much on documenting how they meet accreditation criteria and minimum standards by box-ticking their way to a purchased credibility while paying too little attention to improving their systems and educations 12. Certainly, key performance indicators are important measures of the quality level but they display little evidence of the reasons for the results, including whether the results are due to internal or external factors of the education program 13. Thus, it has been seen that quality development is often obstructed by the national QA systems that in some cases are costly and offer no real evidence of overall quality improvement 14. In order to avoid these possible pitfalls of external quality assurance, including accreditation, DTU has integrated the internal quality enhancement system in the external accreditation process. This integration transforms quality assurance into one of the activities which supports the development of the educations and the understanding more widely among university staff of the prerequisites supporting good and effective student learning and thereby enhancing the quality of the educations. 3. QUALITY IN HIGHER EDUCATION As mentioned previously, quality of education is one of the most difficult areas to measure due to the complex nature of educational situations and learning processes 15. In order to succeed in ensuring quality in HE it is crucial to identify the concept of quality in this context and to specify what is and what produces high quality in education. Even though education is of a complex nature, an increasing amount of research in the area of teaching and learning in HE points out important factors crucial to creating quality. The overall objective in HE is to create the best possible framework for student learning, hence to create quality in learning. If this framework is optimised the students most likely develop the competences which will turn them into highly qualified academics and engineers. 3

4 In overall terms the quality in education is produced in the meeting between the students, the scientific content, the teacher, the planned teaching activities, and the teaching culture and traditions of the university 16. Understanding the processes in this meeting is imperative in order to enhance quality in HE. Most crucial is how the teachers and directors of study programs plan for support of student learning in an effective way. One of the basic perquisites for obtaining such effectiveness is when learning objectives, teaching and assessment methods are aligned 17, and when students are active in their learning. If these conditions are met the students are likely to develop a deep strategy towards learning which is one of the key components in obtaining quality in HE 18. Having described what produces high quality in HE it seems rather difficult to measure this quality of learning. A central question is when to measure? It is very difficult to know exactly when learning occurs; this may even be after the student has graduated. Grading is often taken as a measure of quality but unfortunately the assessment results are influenced by many factors that question its validity as a quality measure of learning 19. Another important element is developing the understanding of student learning within the HE institution. If quality is to be enhanced, every single part of a HE institution needs to develop an awareness of how to support student learning effectively and efficiently 20. This awareness can be developed by professionalising the staff qua different activities such as teacher training programs, rewarding teaching competences and enhancing dialogue about teaching and learning 21. One of purposes of the internal quality enhancement system at DTU is such a faculty and organisational development helping the university staff to do the right things right by making quality a much more visible and concrete subject to work with. 4. THE INTERNAL QUALITY ENHANCEMENT SYSTEM 4.1 Purpose and intentions The internal education evaluation system is part of DTU s overall quality assurance activities within teaching and education. It has been developed in order to achieve three goals: To further development of the educations by facilitating the exchange of experience in order to ensure that the students receive education of excellent quality at DTU, to generate feedback on the implementation of the three plus two (plus three) degree structure, and, last but not least, to strengthen faculty and organisational development in order to enhance a deeper understanding of what is creating quality in educations and student learning. On the basis of the initial experiences with the system, a fourth goal has been added, namely to prepare the organisation and in particular directors of programs, chairmen of study boards, and individual teachers for the external evaluations made by ACE Denmark. The Plan-do-check-act-circle defining the continuous quality improvement 22 then constitutes a double loop process shown in figure 1. Figure 1. Continuous Quality Improvement circle according to Deming 4

5 4.2 The process This system implies that two education programs function as each others evaluation panels on the basis of a self evaluation report written by the directors of study programs responsible for the educations. The directors of study have a key role in the process writing the self evaluation report and analysing their education on basis of the specified evaluation criteria, the result of a focus group interview with students and statistics about study results. In this work they involve other teachers and students, and they are also responsible for assembling the panels. The evaluation is initiated and facilitated by staff from the central DTU study administration and from LearningLab DTU. Educational consultants provide feedback on the report as preparation for the evaluation meeting and facilitate the meeting in order to qualify the discussions and focus on the important prerequisites for student learning in HE. The Dean of education is attending the meeting. LearningLab DTU produces the final report, which is discussed at a meeting with the Dean of education and the directors of study programs. At this meeting, the important potentials for development are identified to be followed up on. 4.3 The criteria The criteria used in the evaluation are developed on the basis of theory on teaching and learning in HE and they are aligned with international concepts such as ABET, EUR-ACE but also with the criteria used by ACE Denmark. The criteria are descriptive and explain to the teachers, directors of study programs and students what important factors that enhance quality in higher education and strengthen the possibilities for the students to learn in an efficient way. The purpose is to generate a more qualified analysis Progression and coherence The introductory text explain how progression and coherence can be achieved in order to make certain that all the different elements in the education program develop the intended competences among the students in a structured way. Requested to describe is the students professional progression, how the objectives of the education are achieved, and what are the most important challenges and possibilities of progression and coherence in the education Teaching methods and learning This criterion focuses on the concept constructing alignment 23 and on the teaching and assessment methods. The introductory text focuses on how the quality in teaching is a result of the interaction between the teacher, the students and the course content 24. The report must contain a description of the teaching and assessment methods that are used in the various courses in the education; the relation between teaching methods, assessment, learning objectives and the competence profile for the whole education; and the most important challenges and possibilities with regard to methods and learning Development of teaching competences In order to enhance student learning, teachers need competences in teaching and learning, enabling them to make the right choices to support student learning efficiently. The director general of the departments who has the overall responsibility for staff development must describe activities in this area including how many resources should be spent on teaching development. The director of the study program describes how the professional and teaching competences among the teachers contribute to realise the goals of the competence profile for the whole education and analyses the most important challenges and possibilities with regard to the development of teaching competences Evaluation of teaching and learning This criterion focuses on the overall quality assurance activities at DTU including various evaluations of courses and educations. These activities generate important information on various subjects and identify potentials for development. The responsibility for follow up on the different evaluations lies with different stakeholders involved in the educations. The report must contain a description of how following up is done, what initiatives have been launched on the basis of the evaluations and how the results are applied, and if the evaluations provide relevant and useful 5

6 knowledge. Finally the most important challenges and possibilities with regard to evaluation of teaching and learning must be described Research based education and scientific methods This criterion focuses on how the education is research based and how to guarantee that the students are trained in the scientific methods of engineering. It is imperative that the teaching is planned and conducted by active researchers and that the content of the program is based on relevant research. The report must contain a description of how the education is research based, how this is reflected or realised in the teaching and how the students benefit from this. 4.4 Underlying ideas and prerequisites The evaluation system incorporates some of the methods used in developing teaching competences in HE. The main idea is to contribute to strengthening a learning community at DTU when it comes to teaching and learning 25. By creating opportunities for teachers to discuss teaching and learning, a common responsibility for the educations is enhanced. With support from the descriptive criteria, the teachers develop a common language and understanding of teaching and learning, including what creates good quality in engineering education. The systematic analysis made by the directors of study programs on the basis of the criteria enhance their understanding of the education and the factors that lead to quality in HE. This is the same learning process that happens when writing and working on a Teaching Portfolio 26. There are other important prerequisites in order for the system to succeed. First of all an atmospher of confidentiality and mutual trust is created. This is probably a result of the main focus on development and potentials for development and explicit absence of control. The study administration and the LearningLab DTU consultants offer support and consultancy while it is the task of the directors of study programs to be critical. Sharing and discussing challenges and problems with colleagues and the dean of education call for this appreciative approach and is supported by the fact that the assessment largely is left to the participants themselves. They are also the ones who do the descriptions and analysis in order to further develop their own and their peers educations. Secondly the concept of quality is qualified and well defined according to this specific context qua the criteria and the introduction texts. Hence it is clearly specified what is meant by quality in rather operational and process oriented terms which makes it possible for the involved stakeholders to understand what is meant by quality in HE and to be concrete and to work systematically. 5. EXPERIENCES AND RESULTS So far three pilot evaluations have been conducted involving a total of six bachelor programs. The results so far indicate that the system is working well and fulfils its purposes. The directors of study program involved describe that they are gaining a clearer and more qualified idea about the conditions for learning in the education. Problems and challenges are becoming more transparent and thereby easier to solve. This is a result of writing the self evaluation report and from the discussions during the meetings. The evaluation meeting were teachers and students participate contribute to an increased willingness among many teachers to develop their courses. This is also described as a result of the criteria that explain to the teachers what can be done to enhance student learning and quality. They point out the systematic, open and qualified discussions about the education and learning as one of the biggest benefits. Also the role of the dean of education in the system is beneficial providing the directors of study program with an effective tool in their discussions with the teachers and director generals of the departments about possible development areas in the education. Furthermore, they describe that they experience increased awareness of what enhance the quality in student learning as a result of the evaluation and that external accreditation of the education seems more manageable in the sense that they have tried a similar situation and prepared an analysis of the education. 6. CONCLUSIONS In this paper we have described the internal quality enhancement system and analysed its immediate impact on the quality of the teaching and learning at DTU. We have shown that the internal evaluations prepare for external 6

7 accreditations and that the latter are strengthened given that a quality culture is evolving at DTU and that quality development has been launched. The apparent success of the system relies on various conditions. The concept of quality has been qualified and defined qua the criteria and the introduction texts making it more operational. This helps the staff to do the right things right. Describing and reflecting on the education according to the predefined criteria enhance faculty understanding of how to improve quality, thus working more as faculty development than quality assurance. The whole system helps directors of study programs to identify and facilitate adjustments and developments in their educations. And, last but not least the system focus on measuring more qualitative aspects and therefore generates information on how and why the quality is produced and enhanced rather than measuring quality on the basis of key performance indicators that display little evidence of the reasons for the results. A thorough evaluation of the system, the results and the effectiveness remains to be conducted. We intend to collect data on the impact and experiences among the participants more systematically in order to assess the benefits. The criteria for this assessment have to be rather soft due to the fact that measuring the outcome of the evaluation process via enhanced student learning is very difficult cf. the previous section on quality in HE. Thus the approach will be dialogue oriented focusing on the participants assessment of the process consisting of their personal experience of lessons learned and specific improvements or outcomes. The overall success criterion then, is whether the evaluation process contributes to organisational learning and tangible developments. And so far the internal quality enhancement system has proven its worth. Acknowledgment The authors wish to thank Patricia Staaf and others at Malmö University, and Viveca Serder, Eslöv municipality, former employee at Malmö University who generously shared their ideas and experiences from a similar quality enhancement system. Their ideas and experiences have greatly inspired the development of the DTU system. References 1 P. Dahler-Larsen (2008), Kvalitetens beskaffenhed (in Danish), Syddansk Universitetsforlag, Odense 2 P. Dahler-Larsen (2003), Monitering af undervisning (in Danish), In: P. Dahler-Larsen and H.K. Krogstrup (edt.), Tendenser i evaluering, Syddansk Universitetsforlag, Odense 3 C. Rozsnyai (2003), Quality Assurance Before and After Bologna in the central and Eastern Region of the European Higher Education Area with a focus on Hungary, the Czech Republic and Poland : European Journal of Education, Vol. 38, No C. Rozsnyai (2003), Quality Assurance Before and After Bologna in the central and Eastern Region of the European Higher Education Area with a focus on Hungary, the Czech Republic and Poland : European Journal of Education, Vol. 38, No L. Harvey (2005), A history and critique of quality evaluation in the UK : Quality Assurance in Education, Vol. 13 No. 4 6 M. Jeliazkova and D.F. Westerheijden (2002), Systemic adaption to a changing environment: Towards a next generation of quality assurance models : Higher Education, M. Jeliazkova and D.F. Westerheijden (2002), Systemic adaption to a changing environment: Towards a next generation of quality assurance models, In: Higher Education, 44..; T. Saarinen (2005), Quality in the Bologna Process: from competitive edge to quality assurance techniques : European Journal of Educatioon, Vol. 40, No. 2.; G. Haug (2003), Quality assurance/accreditation in the Emerging European Higher Education Area: a possible scenario for the future : European Journal of Education, Vol.38,No3. 8 M.C. Van Der Wende and D.F. Westerheijden (2001), International Aspects of Quality Assurance with a Special Focus on European Higher Education : Quality in Higher Education, Vol. 7, No T. Saarinen (2005), Quality in the Bologna Process: from competitive edge to quality assurance techniques : European Journal of Educatioon, Vol. 40, No G. Haug (2003), Quality assurance/accreditation in the Emerging European Higher Education Area: a possible scenario for the future : European Journal of Education, Vol. 38, No Berlin Communiqué (2003), Realising the European Higher Education Area. Communiqué of the Conference of ministers responsible for Higher Education in Berlin on 19 September L. Harvey (2005), A history and critique of quality evaluation in the UK : Quality Assurance in Education, Vol. 13 No P. Dahler-Larsen (2003), Monitering af undervisning (in Danish): P. Dahler-Larsen and H.K. Krogstrup (edt.), Tendenser i evaluering, Syddansk Universitetsforlag, Odense 14 EUA (2007) Trends V, Universities shaping the European Higher Education Area. Brussels. 7

8 15 P. Dahler-Larsen (2003), Monitering af undervisning (in Danish), P. Dahler-Larsen and H.K. Krogstrup (edt.), Tendenser i evaluering, Syddansk Universitetsforlag, Odense 16 W. Jank and H. Meyer (2006), Didaktiske modeler grundbog i didaktik (Danish translation of Didaktische Modelle), Gyldendal, Copenhagen 17 J. Biggs and C. Tang (2007), Teaching for Quality Learning at University, Open University Press, Berkshire; P. Ramsden (1991), Learning to teach in Higher Education, Routledge, New York 18 F. Marton & S. Booth (1997), Learning And Awareness, Lawrence Erlbaum Associates, INC, New Jersey 19 S. Brown and A. Glasner edt. (2000), Assessment Matters in Higher Education, Open University Press, Buckingham 20 J. Bowden and F, Marton (1998), The University of Learning Beyond Quality and Competence, Kogan Page, London 21 E. L. Boyer (1990), Scholarship Reconsidered - Priorities of the Professoriate, Princeton University Press, New Jersey 22 D.W. Edwards (1986), Out of the Crisis, MIT Press 23 J. Biggs and C. Tang (2007), Teaching for Quality Learning at University, Open University Press, Buckingham 24 W. Jank and H. Meyer (2006), Didaktiske modeler grundbog i didaktik (Danish translation of Didaktische Modelle), Gyldendal, Copenhagen 25 E. Wenger (1998), Communities of practice. Learning, meaning and identity, Cambridge University Press, New York 26 P. Seldin (2004), The teaching portfolio: A practical guide to improved performance and promotion/tenure decisions, Anker Publishing Company, Bolton 8