Finnish Higher Education Evaluation Council FINHEEC Dr. Seppo Saari, docent www.kka.fi Seminar South-Africa October 2006
Finland in the World 2
3
Outline FINHEEC's role Background for QA in Finland Audit framework Feedback from Audits 4
FINHEEC (1/5) Duties (Decrees 1320/95 and 465/98) Initiate evaluations of higher education and promoting its development Engagement in international evaluation cooperation Promote research on evaluation of higher education; and Evaluation and approval of professional courses offered by higher education insitutions, entering of courses into a register as stipulated in Article 14 of the Decree on the Higher Education System, and maintenance of such a register. 5
Aims of FINHEEC (2/5) The most fundamental aim of FINHEEC is a long-term development of higher education through evaluations. For this reason, FINHEEC: Supports higher education institutions while they design their own quality assurance and evaluation systems; Supports the HE institutions own evaluation work Financial aid for projects Benchmarking; Produces national data enabling international comparison of higher education institutions for policy makers, students, trade and industry. 6
Evaluations (3/5) 1. Evaluations focusing on individual universities (20)/polytechnics (29) Institutional evaluations of both HE sectors Evaluations of quality work in polytechnics ------> to be substituted by the auditing of quality assurance systems of higher education institutions 2. Programme evaluations PhD education in Universities Master s programmes Media and communications studies Open university education Regional impact of HEs in the Lahti region Selection of students into universities Civil engineering 7
Evaluations (4/5) 3. Education policy and other thematic evaluations 4. Student as a citizen in HEIs Education and themes that have social or educational importance or that are rapidly expanding, developing or problematic 5. Evaluation and registration of professional courses (accreditation) 6. Evaluation assignments implemented on the order of universities/polytechnics and the Ministry of Education as a separate commercial service 8
Evaluations (5/5) 6. Selection of quality units and centres of excellence Polytechnics Quality units of education (2000, 2002, 2005) Regional centres of excellence of polytechnics (2001, 2003, 2006) Universities Quality units of basic and/or continuing education (1997, 2000, 2003, 2007) Quality units of adult education (1997, 2000, 2003, 2007) 9
Background for QA and audits (1/3) International development in the higher education sector Bologna declaration 1999 and the objective to create coherent European higher education area by 2010 Communiqué of the Conference of Ministers responsible for Higher Education in Berlin 2003: The quality of higher education is at the heart of the setting up of a European Higher Education Area (development of quality assurance at institutional, national and European level) The need to develop mutually shared criteria and methodologies on quality assurance 10
Background for QA and audits (2/3) By 2005 national quality assurance systems should include: A definition of the responsibilities of the bodies and institutions involved Evaluation of programmes or institutions, including internal assessment, external review, participation of students and the publication of results A system of accreditation, certification or comparable procedures International participation, co-operation and networking 11
Background for QA and audits (3/3) The aims of the quality audits are to create an effective national quality assurance system demonstrate the higher education institutions preconditions for quality in their activities for the national and international stakeholders 12
Quality Assurance in Higher Education (1/3) Quality assurance of the Finnish higher education is constituted of three elements: 1. National HE policy; 2. National evaluations; 3. HEIs' own quality assurance 13
Quality Assurance in Higher Education (2/3) National quality assurance signifies for the procedures, processes or systems, which aim at assuring the quality of higher education at a national level The main responsibility for the quality of education, R&D, scientific research and other operations lies on the individual higher education institutions 14
Quality Assurance in Higher Education (3/3) The Ministry of Education instituted a working group in 2003 to analyse the present stage and development requirements for quality assurance in Finnish higher education The working group proposed that in view of the communiqué of the ministerial meeting in Berlin, Finland will implement a system of evaluating the quality assurance of universities and polytechnics 15
Requirements for QAsystems (1/2) The quality assurance systems should a) meet the developing quality assurance criteria of the European Higher Education Area; b) be part of the operational steering and management system; c) cover the entire operation of the higher education institution; 16
Requirements for QAsystems (2/2) d) be interrelated as part of the normal operations of the higher education institution; e) be continuous; f) be documented; and g) enable the participation of all members of the higher education community in quality work 17
Audit Framework (1/5) Each higher education institution has a quality assurance system that best suits its own operations and set aims The objective is to audit the quality assurance systems assuring the quality of the higher education institutions' operations, not the operations themselves or their outcomes 18
Audit framework (2/5) The main responsibility of the quality assurance systems remains in the higher education institutions The approach in quality audits is fitness for purpose quality assurance systems are fit for purpose in advancing the institutions mission and aims Continuous quality development and enhancement 19
Audit framework (3/5) The aims of the quality audits are to: Support higher education institutions in the development of quality assurance Evaluate, how the quality assurance systems function in relation to the institutions aims Quality assurance systems include the procedures processes and mechanisms by which the institutions manage and develop the quality of the higher education Show for the national and international stakeholders the quality level of higher education in Finland 20
Audit Framework (4/5) The general aim is the continuous developing of quality The audits of quality assurance systems will replace former evaluations of higher education institutions carried out by FINHEEC: institutional evaluations on both higher education sectors and audits of quality work in the polytechnic sector 21
Audit Framework (5/5) Consequences of audits two options Audit processes and methodology: self-evaluation documents are consisted of existing documents (not prepared just for the audit) audit visit criteria for the members of the audit panel public audit report 22
Quality assurance system level Quality assurance whole Setting of objectives Strategic planning Leadership and management Resourcing Quality assurance of basic tasks Research/ R&D Support and service functions Degree education Societal interaction, regional development (continuing education, open HEI teaching) HR development 23
What is being audited? Coverage, functionality, openness and communicativeness of the QA system 24
Auditing targets 1. Objectives, overall structure and internal coherence of the QA system 5. Interface between the QA system and management/steering 2. Documentation including the formulation of Q policy and the definition of procedures, actors and responsibilities 3. Comprehensiveness of QA focusing on basic tasks and support and service functions degree education research/r&d - interaction with and impact on society; and contribution to regional development support and other services staff development 4. Participation of staff, students and external stakeholders in QA 6. Relevance of and access to, QA information 7. Relevance of, and access to, quality assurance information for external stakeholders 8. Efficiency of QA procedures and structures and their effect on the development of activities 9. Use of information produced by the QA system as tools for quality management and enhancement in education and other activities 10. Monitoring, evaluation and continuous development of the QAsystem 25
Scaled criteria of targets The selection of the criteria is based on their relevance for the evaluation of the functioning of the HEI s QA system, leaving the HEIs scope of movement and the freedom of choice in the use of their own QA system The evaluation of the QA system is based on a scaled set of criteria focusing on four levels of development (absent, emerging, developing and advanced) Rather than being mere descriptions of operations, the evaluation criteria go deeper The HEI should be able to prove the functioning of the QA system and convince the external panel of its impact. Process criteria - content criteria Process criteria: good/bad quality itself is not defined, HEI responsible for quality definition, audit focuses on processes Content criteria: high-quality operations are defined based on the criteria. 26
When is quality assurance system sufficient? Premise is constituted by the objectives and criteria Evaluation of sufficiency is based on: HEI s own justifications and evidence Core questions: - how can the HEI know that its QA is working and how can the HEI prove it? - is the QA system serving the continuous enhancement (in the line with the strategy and objectives) Expertise and experience of the audit group members (HEI representatives, student representatives, practical employment sector = work life representatives) Quantity & quality of the QA procedures? 27
Targets used in auditing 1. Objectives and overall structure of the QA system and its internal coherence What are the elements included in the QA system? Does it comprise all core areas of operations? Is there coherence between the various elements? Are the actors aware of the links between the various elements? How does the system work in practice? Has the system been cascaded down to the organisation? 2. Quality-related documentation, including the formulation of quality policy and definition of activities, actors and responsibilities How and where have the functions, actors and responsibilities of the QA system been defined? Are the documents easily accessible? Are factual operations and division of labour in line with descriptions in documentation? How efficient is the division of responsibilities? 28
3. Comprehensiveness of QA in terms of different activities and processes relating to basic mission Targets used in auditing 3 a) Degree education 3 b) Research / R&D 3 c) Interaction with and impact on society; contribution to regional development 3 d) Support and other services (e.g. library and information services, career and recruitment services, and international services) 3 e) Staff development Does QA comprehend all areas of planning, implementation and evaluation of education? Does QA comprehend all areas of research/r&d? Does QA comprehend all areas of societal interaction and impact and regional development? Does QA comprehend all areas of support and service functions? Does QA comprehend all areas of HR development? 29
Targets used in auditing 4. Participation of staff, students and external stakeholders in QA In what way do the members of the HEI community participate in QA? Are they committed and active? How is participation promoted and supported? students teaching staff support service staff researchers administration management What is the role of external stakeholders in the operation of the QA system? 30
Targets used in auditing 5. Interface between the QA system and HEI leadership and management In what way have the HEI objectives been considered in the construction of the QA system? What is the link between the operation of the QA system and strategic planning? In what way is the information provided by the QA system utilised in the steering of operations, follow-up of results and development? What are the proofs of such utilisation of information? What are the links between the QA system and resourcing decisions? In what way is the QA system used as a tool for the steering of operations? Is the HEI able to use the information produced by the QA system to form an overall picture of the quality of its operations? Management commitment? 31
Targets used in auditing 6. Relevance of and access to information produced by the QA system within the HEI 7. Relevance and availability of information produced by the QA system from the perspective of external stakeholders In what way has the planning of the QA operations focused on the relevance of the information produced by the QA system from the perspective of the HEI s internal actors? Have the information needs of different actors been taken into account? Are the actors aware of the operation of the QA system and its core results? Are communications active and frequent? Have the external stakeholders been defined? In what way has the planning of the QA operations focused on the relevance of the information produced by the QA system from the perspective of cooperation partners and external stakeholders? Are the communications active? Are they targeted at precisely defined groups? 32
Targets used in auditing 8. Efficiency of QA procedures and structures and their effect on the development of activities Functioning and impact of the QA procedures and structures (cfr. items 1 and 3a-e) Does the QA system include modes of operation and structures that promote new ideas and change? What are they like? In what way do they support the development of operations? Does the operating culture support innovativeness? In what way does the QA system help to identify insufficient quality? 9. Use of information produced by the QA system as a tool for quality management and enhancement in education and other activities In what way and how systematically is the information produced by the QA system used for the management and enhancement of the quality of the basic tasks? Can the HEI show proof of that? 33
Targets used in auditing 10. Monitoring, evaluation and continuous development of the QA system What is the overall view of the HEI management, staff and students of the operation and impacts of the system? What are the means to follow and develop the system? 34
Degree education - examples of processes and procedures Planning, implementation and evaluation of degree education Planning and approval of new education and training Application procedures, student selection, training need Curriculum process Teaching methods Learning environments Practical training Thesis Assessment of learning results Course of study, degrees completed, drop-outs: Education-focused follow-up and evaluation procedures and their development 35
Examples of QA procedures included in degree education (3a) Instructions and documents: Planning of education, student selection, study guidance, implementation of education (e.g., teaching methods, training, theses), evaluation and development Gathering and use of evaluation data and feedback: assessment of learning results, student feedback system, staff feedback questionnaires, feedback from practical employment sector or from alumni, gathering and use of statistics (e.g., progress of studies, degrees, drop-outs), performance appraisals, feedback sessions, self-evaluations by degree programmes/departments, internal audits or cross audits, external auditing Other procedures supporting QA and development: - ways of organising operations: participation of students and work-life exponents in curriculum development, innovative new ways to construct and perceive study entities (incl. those improving understanding and communicativeness), organisation of steering discussions, construction of personal study plans (HOPS) - R&D projects: development of teaching methods or new evaluation methods 36
Examples of QA procedures included in research/r&d (3b) Instructions and documents related to research/r&d as well as gathering of evaluation and feedback data Linking research/r&d with teaching: students in projects (theses, practical training), students in projects (e.g., as trainers, experts, postgraduate students) Participation by practical employment sector exponents and their influence through advisory committees Linking of research topics and projects to the overall strategy of the HEI 37
Examples of procedures related to societal interaction, impact and regional development (3c) Open university/polytechnic teaching and continuing education: operating instructions/documents as well as gathering of evaluation and feedback data (incl. training needs, marketing, student selection, implementation of teaching) Cooperation between HEI and external stakeholders in implementation of teaching and research/r&d: instructions, documents as well as gathering of evaluation and feedback data (numbers of persons or organisations, feedback from external stakeholders) Anticipating training needs: regional development studies, gathering of forecast information 38
Examples of QA procedures included in support and service functions (3d) Instructions, documents, gathering of evaluation and feedback data as well as QA and development data regarding, for example, the following support and service function processes: - library and information services - career and recruitment services - international services - information management services - marketing and communications - administrative services (incl. financial administration, real estate services) 39
Examples of QA procedures related to HR development (3e) HR planning and resourcing: operating instructions and selection criteria for recruitments Competence and level of education of staff: additional and continuing education of staff (e.g., pedagogic training), organisation of various thematic and development meetings and seminars Resources and job satisfaction of staff: gathering and use of feedback, activities for maintained working ability, performance appraisals Leadership and management skills: additional and continuing education 40
Auditing targets and criteria 1. Objectives, overall structure and internal coherence of the quality assurance system ABSENT The HEI has no QA procedures in place relating to its activities. EMERGING There are QA procedures relating to some HEI activities, but they are neither systematic in structure nor interlinked. DEVELOPING QA covers many of the HEI's activities and the QA procedures form a fairly efficient system. ADVANCED QA covers all or nearly all activities, and QA procedures processes form a dynamic whole. 41
Auditing targets and criteria 2. Documentation, including the formulation of quality policy and definition of procedures, activities, actors and responsibilities ABSENT EMERGING Quality policy, activities, actors and responsibilities have not been defined or documented. DEVELOPING Activities, actors and responsibilities are clearly, comprehensibly and concretely defined, and the documentation is easily available to all. The organisation is well-designed. The definition and documentation of responsibilities and activities included in the QA system are inadequate, and the procedures are inadequately organised. ADVANCED The activities and division of labour manifestly comply with the documented processes. The organisation is extremely well-designed and reinforces QA. 42
Auditing targets and criteria 3. Comprehensiveness of QA in terms of different activities and processes relating to basic mission ABSENT No QA in the activities and processes relating to basic mission. EMERGING The system covers isolated activities and processes mainly relating to degree education. DEVELOPING The system covers many activities and processes relating to basic mission. ADVANCED In the main, the system covers activities and procedures relating to the basic mission. 43
Auditing targets and criteria 3 a) Degree education ABSENT No QA in degree education. EMERGING QA covers some isolated aspects of the planning, implementation and evaluation of degree education. DEVELOPING QA covers several aspects of the planning, implementation and evaluation of degree education. ADVANCED QA covers all salient aspects of the planning, implementation and evaluation of degree education. 44
Auditing targets and criteria 6. Relevance of and access to information produced by the QA system within the HEI ABSENT QA does not cater for internal stakeholders and information is not communicated within the HEI. EMERGING The production of information is not systematic and does not sufficiently cater for internal stakeholders. DEVELOPING The activities and key results of the QA system are known to internal stakeholders. The system produces information relevant to them. ADVANCED The operation of the QA system is overt and transparent. Internal communication relating to QA is active and information is systematically communicated and targeted to different parties within the HEI. The relevance of information to internal stakeholders is an important consideration in the planning and continuous development of the QA system. 45
Two levels of focus in auditing Starting point is the HEI s own system, developed on the basis of its premises. Through this approach, it is possible to promote the development of HEI s own quality culture, and promote an active and comprehensive participation of the HEI, and enhance the sense of QA ownership among the HEI actors. Object of audit constituted by QA processes and procedures (fitness for purpose perspective) - audit does not address HEI s objectives, operative contents and results per se. Two levels of focus QA as a whole QA of the basic tasks 46
Challenges Combining the two levels of focus and related challenges (cfr. total quality and quality of degree education). How to address the real quality of education? Quality cultures in administration and operative control as well as on the grass-root level: understanding and combining different perspectives (cfr. objectives of second visit day) Links between the QA procedures included in the HEI s basic tasks Sufficient scope and depth of auditing, especially as concerns large HEIs, is a challenge for the audits. Audit efficacy is influenced by: relevance of auditing criteria auditing methods (scope and depth of audit material, implementation of audit visit) 47
Three main areas in recommendations Functionality of QA system Comprehensiveness of QA systems Efficiency of QA system 48
Thank you for your attention! 49