Quality Assurance in Degree Programmes

Transcription

1 Quality Assurance in Degree Programmes Monitoring and Evaluation in Higher Education Mapping the Quality Debate Bologna Process, Berlin Communiqué ESG New Public Management Internal Quality Assurance 1

2 Framing Quality Assurance ESG Could serve as a road map for developing qualty policies at institution level. Berlin Communiqué (2003) on QA: mutually shared criteria and methodologies agreed set of standards, procedures and guidelines Framing Quality Assurance ESG Problems and Obstacles: - Implementation: How to apply the ESG in a concrete QA system? - Responsibilities: Who is in charge of implementing QA procedures? - Trust and Support: Quality Assurance or Quality Controlling? 2

3 Agenda: Overview Breaking down the ESG to applicable internal QA procedures Clarifying the Concepts: Evaluation, QA, QM, Monitoring Evaluation Research Putting Evaluation into Practice Workshop Organisation Keynotes Group Work Online Support. Download materials and take part in further discussions: Launch Click Quality Assurance in Degree Programmes: Monitoring and Evaluation Create an account Confirm your account via 3

4 Click here to create your account FIll in your contact details here 4

5 Remaining Questions? Philipp Pohlenz University of Potsdam The President s Office, Quality Assurance Unit Am Neuen Palais 10 D Potsdam, Germany Tel Fax pohlenz@uni-potsdam.de 5

6 Agenda (day 1) Clarifying the concepts: What is evaluation and QA all about? What data to monitor in QA: strengths and weaknesses How to design an evaluation research project? Breaking down the ESG to manageable research questions, Part 1. Clarifying the Concepts Quality assurance, quality management, monitoring, Often used synonymously evaluation Do we need to distinguish? What is distinguishing them? 1

7 Clarifying the Concepts Quality Assurance (QA) Used in the most general sense Any activity in an organisation that aims at securing the quality of the organisation s provision Clarifying the Concepts Quality Management (QM) Set of activities or routines, continuously implemented to achieve quality goals Quality is the ruling philosophy of the organisation All members of the organisation are committed to achieving quality goals All members are involved in quality strategy Top-down planning of quality-policy by leadership E.g. the EFQM Excellence Model 2

8 Clarifying the Concepts The EFQM Excellence Model Clarifying the Concepts Monitoring as a tool for QM Management information on progress and achievement Descriptive intent: where is the programme relative to its targets? at any given time. E.g. how many graduates per academic year? Systematic collection of data on specified indicators Links activities and resources to objectives 3

9 Clarifying the Concepts Evaluation as a tool for QM Management information on progress and achievement Analytic intent: why is the programme underachieving relatively to its targets E.g. what are the reasons for student-dropout? Research activity; inquiry on specific issues Assesses causal contribution of activities to results Clarifying the Concepts Defining Evaluation: Evaluation is the systematic inquiry of the value or merit of a programme, activity, project Research findings, conclusions, and recommendations for future action need to be comprehensibly based on empirical data and evidence. A variety of research methods, mainly from the social sciences are being applied. Therefore it follows: evaluation research is applied social science research. 4

10 Clarifying the Concepts Quality Assurance Quality Management Using Management Information Monitoring Evaluation Creating Management Information 5

11 What Data to Monitor? Content: what information do we need? what data on which indicators deliver the respective knowledge? Quality: Clear = precise and unambiguous Relevant = appropriate to the subject Economic = available at reasonable cost Adequate = sufficient basis to assess Monitorable = amenable to independent validation What Data to Monitor? Performance Indicators in HE Comparison within and between HE institutions budgeting Accountability of stakeholders Competition for public funding (NPM) Objectives: Decentral planning based on management information; development of profiles Incentives for achievement of public goals (e.g. number of graduates per ac. year) 1

12 What Data to Monitor? Performance Indicators in HE: advantages Output-oriented steering of universities, expost assessment of achievement Economically efficient performance Management by objectives What Data to Monitor? Performance Indicators in HE: problems Provokes adjustments of strategic planning to external expectations instead of internal objectives (e.g. shutting down programmes of high academic interest and low financial efficiency; awarding academic degrees to unfit candidates) 2

13 What Data to Monitor? Indicators in use in programme monitoring Number of students in regular study time Capacity utilisation (places : applications) Incentive: attracting students enrolment is being rewarded. What Data to Monitor? Indicators in use in programme monitoring Graduates : graduates of past periods Graduates : students in a specific period Graduates in regular time : overall graduates Incentive: increasing the number of graduates, decreasing student drop-out, and cutting down study times is being rewarded. 3

14 What Data to Monitor? Problems: Some indicators measure inputs rather than outputs, but claim to indicate outputs, results, achievements Input information tells more about the conditions under which a university works than about its achievement What Data to Monitor? Problems: E.g. the number of students does not necessarily tell about a university s attractiveness, since students choose their university due to individual reasons. E.g. student drop-out is not only caused by bad working conditions, but as well by individual reasons which are beyond a university s influence. 4

15 What Data to Monitor? Clear = precise and unambiguous Relevant = appropriate to the subject Economic = available at reasonable cost Adequate = sufficient basis to assess Monitorable = amenable to independent validation What Data to Monitor? Student drop-out and the statistics. Semester 1 Semester students 80 students = 20 dropouts??? We just don t know!! 30 drop-outs + 10 newcomers??? 5

16 What Data to Monitor? Quality of the student drop-out indicator? What can we learn from the data, if they are apparently distorted? What consequence if we base management decisions on such data? How to overcome such problem? What Data to Monitor? Biographical data at the level of each student Cohorts of students and their lifecycles at the university Recognition of migration between programmes (at the same university) Precise numbers for student drop-out. 6

17 What Data to Monitor? Example: Migration of students from Macroeconomics to business administration. Migration out of a study programme usually counts as drop-out. Negative consequences for the Department of Macroeconom. But: distorted indicators What Data to Monitor? Cohort 1999/2000; Macroeconomics 7

18 What Data to Monitor? Conclusion: Performance indicators give valuable information on the current status of a programme. However, they should not be overestimated as they mostly describe the input-side. Limitations need to be recognised. What Data to Monitor? Conclusion: Indicators do not provide explanations, they need to be complemented by evaluations. Input Output Outcome Students Graduates Employability Indicators Evaluation 8

19 How to Design Evaluation? Definition: Evaluation is the systematic inquiry of a programme s value or merit. Findings, conclusions, and recommendations need to be comprehensibly based on empirical data. [German Evaluation Society] How to Design Evaluation? Evaluation research is supposed to Analyse effects of programmes or actions on stakeholders, participants, policies, Contribute to improvements of processes and performances Increase value of actions 1

20 How to Design Evaluation? Evaluation research analyses social processes (e.g. learning and teaching) Evaluation applies empirical research methods Evaluation is applied social science research. How to Design Evaluation? Social Science Research 1. Identifying a research problem 2. Theory-building 3. Operationalising the research question 4. Field work (sampling, data gathering) 5. Analysis 6. Publication Evaluation Research 1. Status analysis 2. Developing a programme theory 3. Research design 4. Data gathering 5. Analysing 6. Policy Making 7. Re-evaluating (evaluation circle) 2

21 How to Design Evaluation? Development and implementation of an action strategy Publication of results Data gathering and analysis Status report, problem identification Development of research tools (questionnaires, etc.), research plans (sampling) Planning and target formulation, quality dimensions Choosing research methods Development of research questions How to Design Evaluation? Problem identification The number of graduates in regular time has decreased during the last five years! Employers complain about a low employability of our graduates! How can this be explained? What action needs to be taken? 3

22 How to Design Evaluation? Planning and target formulation Who will be the audience of the evaluation report? What information is needed? Time schedule Quality dimensions, research questions What is employability? Operationalisation of abstract concepts, translation into research questions. How to Design Evaluation? Choosing research methods How can the questions best be addressed? Interviews (narrative interviewing) Focus group discussions Expert interviews Content analysis Meta analysis (statistics) 4

23 How to Design Evaluation? Research plan Who to address? Employability of graduates: Graduates, employers, parents Statistics on the labour market Depends on content questions and practical issues (available funding for the research Sampling Representative sampling? Explorative? How to Design Evaluation? Data gathering and analysis Field work, Pretest? Postal dissemination of questionnaires, online surveys? Proper addressing of interviewees Data security Data processing (non-response) 5

24 How to Design Evaluation? Publication of findings Who is the target audience, what purposes to serve with the report? Internal discussions Information of the public, marketing accountability How to Design Evaluation? Development of an action strategy Measurements are taken on the base of the evaluation results Low employability in the graduates results in a programme on soft skills Re-evaluation of the refined programme Has the programme resulted in an increased employability? 6

25 How to Design Evaluation? Standards for good evaluation Usefulnes Goal-oriented, transparent, timely Feasibility Realistic planning, appropriate means and methods Fairness Unbiased implementation, holistic How to Design Evaluation? Standards for good evaluation Accuracy Clear identification of the issue under evaluation Valid, reliable data Documentation of implementation and sources of information Comprehensible conclusions Mixed methods approach 7

26 Evaluation at Programme-level Three-step procedure Internal self-evaluation Self-report, evaluation findings: statistics, surveys Peer review Site-visit by experts, talks based on the self-report, recommendations for improvements Follow-up measurements Stipulation of measurements and actions to be taken, refinement of curricula, organisational improvements, etc. 8

27 The ESG for QA The Berlin Communiqué (2003): Ministers called upon ENQA to develop an agreed set of standards, procedures and guidelines on quality assurance and to explore ways of ensuring an adequate peer review system for quality assurance and/or accreditation agencies or bodies, and to report back through the Bologna Follow-Up Group to Ministers in The ESG for QA The guidelines for internal QA within higher education institutions: (1.1) Policy and procedures for quality assurance (1.2) Approval, monitoring and periodic review of programmes and awards (1.3) Assessment of students 1

28 The ESG for QA The guidelines for internal QA within higher education institutions: (1.4) Quality assurance of teaching staff (1.5) Learning resources and student support (1.6) Information systems (1.7) Public information The ESG for QA ESG function as the major guideline for QA processes in European higher education. Comparable to sets of standards in other regions worldwide: even though quality in higher education is hard to define, there seem to be agreed dimensions that could serve to describe it. 2

29 The ESG for QA Problems: Formulated abstractly, Set out standards but no actual procedures for QA-routines at programme leve ESG need to be translated into a concrete evaluation design The ESG for QA Breaking down the ESG Standard: Policy and Procedures for QA Guidelines: The policy statement is expected to include: Diagnostic questions 3

30 The ESG for QA Institutions should have formal mechanisms for the approval, periodic review and monitoring of their programmes and awards. The quality assurance of programmes and awards are expected to include: availability of appropriate learning resources; monitoring of the progress and achievements of students The ESG for QA How to investigate the appropriateness of the available learning resources? Ask the students about their view on the available learning resources, consult best practice examples and benchmarks, How to discover progress and achievement? Consult statistics on learning results, ask students about their self-assessment, 4

31 The ESG for QA Let s go! Assignment: Imagine, you are asked by the Rector of your university to establish a monitoring & evaluation system for QA in teaching and learning, based on the ESG. How would you handle this task? 5

32 Gathering Relevant Data Research designs for the inquiry of the diagnostic questions can be Exploratory Descriptive Causal analytic Each requires the application of different (statistical) methods. Gathering Relevant Data Exploratory research Gaining general insight into the field Description of population parameters Statistical measures: mean values, variances Not hypotheses testing Preparation of further research questions If qualitative methods are applied: brainstorming, focus group discussions, etc. 1

33 Gathering Relevant Data Exploratory research Gathering Relevant Data Descriptive research: describes interrelations of parameters (e.g. student achievement and their time budget) Population parameters Correlations Cross tabulation 2

34 Gathering Relevant Data Descriptive research Gathering Relevant Data Descriptive research EcoAware * education Crosstabulation Count EcoAware Total education Total

35 Gathering Relevant Data Descriptive research Correlations, Hypotheses testing The higher the level of education, the lower the BUT: no causal relationships can be inquired. Which variable is cause and which one effect? Independent variable (cause) Dependent variable (effect) Gathering Relevant Data Descriptive research Correlations, Hypotheses testing The higher the level of education, the lower the BUT: no causal relationships can be inquired. Which variable is cause and which one effect? Independent variable (cause) Dependent variable (effect) 4

36 Gathering Relevant Data Causal research Preconditions for the investigation of causal effects: Experimental design: Randomised assignment of experimental and control groups Control of interactive variables Longitudinal research designs: observation of effects over time Advanced methods and research designs required Gathering Relevant Data Example for longitudinal studies: Cohort/Panel design Cohort: group of persons that share an event in the life cycle (age group, graduates of 2002, etc.) Panel: group of persons that is repeatedly interviewed over time. 5

37 Gathering Relevant Data t 1 t 2 t 3 t 4 Cohort Panel Design Cohort Cohort Gathering Relevant Data Online-Panel for web-based panel surveys 6

38 Gathering Relevant Data Online-Panel for web-based panel surveys Gathering Relevant Data Reliability and Validity of the data Basic requirements for good data collection Reliability: do measurements of a variable produce the same results over time and in different groups? Validity: does the instrument (e.g. questionnaire) measure what was intended to be measured? Is teaching performance validly measured by teacher s attractiveness? 7

39 Gathering Relevant Data Reliability and Validity of the data can be endangered by Subjectivity: e.g. student ratings of courses which are influenced by their sympathy for the teacher, Conditions: e.g. comparison of universities that are operated under different circumstances (rural vs. urban, ) Unfair assessments result in resistance against the evaluation process as a whole. Gathering Relevant Data Endangered reliability and validity of the data need to be handled: Analysis: check statistically, e.g. via correlation / regression analysis: what are the determinants of student appraisals? Policy: Make limitations of evaluation findings transparent and discuss with those under evaluation. 8

40 Putting Evaluation into Practice Evaluation findings are supposed to be translated into action. They promote Organisational learning Quality culture Refinement of strategies and routines Putting Evaluation into Practice Organisational Learning Assertion of new knowledge within organisations which results in new routines and structures, independent of individual persons. The resolution of conflicts (as regards interests and allocation) is a precondition for the successful assertion of new routines. Evaluation is policy making and mediation. 1

41 Putting Evaluation into Practice Organisational Learning: examples Adjustment of curricula to changing demands of society and the labour market Restructuring decision making processes in order to improve the involvement of students Putting Evaluation into Practice Organisational Learning: requirements Moderation Communication Clarification of benefits for stakeholders from the introduction of measurements Democratic command structure Mediation and consensus 2

42 Putting Evaluation into Practice Quality Culture: Quality awareness of all members of the organisation Joint action, based on a shared understanding of what quality is Mutual committment to contribute to the achievement of the organisation s quality goals Institutional knowledge of quality (dimension, appropriate policy,..) Putting Evaluation into Practice Quality Culture: instruments Communication and cooperation in a participative command structure (e.g. quality circles which involve all stakeholder groups) Involvement of all stakeholder groups in management decision making processes Assignment of working groups with the moderation of organisational learning processes 3

43 Putting Evaluation into Practice Quality Culture: tasks for the leadership Establishment of conditions (participative leadership style, promoting organisational learning) Transparent management Provision of instruments Establishment of an information system (knowledge management) Putting Evaluation into Practice Introducing Quality Management at Potsdam University: Evaluation procedures since 1998 (student ratings, target agreements, data monitoring, ) From 2007: overarching framework for quality management at all levels: education, research, services, Systematic application of evaluation research and use of data for management and controlling 4

44 Putting Evaluation into Practice Mechanisms for the continuous utilisation of evaluation data Target agreements between leadership and departments, Working units to accompany evaluation procedures checks and balances : committment for quality will be rewarded Putting Evaluation into Practice Counter flow principle Balancing leadership decisions and decentral autonomy of departments Reports, committment for quality goals Departments Leadership Rewards, cooperation, consultation, involvement Target Agreements 5

45 Putting Evaluation into Practice Implementation Leadership sets out overall quality goals (quality charter) Departments set out quality goal at dep. level Departments carry out monitoring & evaluation procedures QA-unit supports departments (data gathering and analysis) Advisory board discusses and evaluates target agreements Knowldege-based action (knowledge management) 6

46 Putting Evaluation into Practice QM-processes in the field of education Learning outcomes (competencies) Evaluation (programmes, courses) Education Accreditation Internationalisation Student assessment 7

47 QA in Practice: Responsibilities Professionalising QA What is a Profession? Describes the work that is done in a specific fields by experts that have undergone a specific training in the field comply with and apply the rules and methods of the respective area of research enjoy the trust of the public because of their specific expertise and proficiency. QA in Practice: Responsibilities QA-activities in higher education are conducted by Psycologists, Sociologists, Economists, (sometimes Historians) There is not a QA-profession by now. 1

48 QA in Practice: Responsibilities Needed: Training programmes in QA, degrees (MA, PhD) Curriculum in the field of higher education: Research Methods Counseling skills Management Capabilities Leadership Organisational Development QA in Practice: Responsibilities Tasks of a QA-officer Variety of assignments require a broad understanding of the field: Supporting departments and faculties with self-assessment and reporting Conducting research Contracting follow-up measurements, observing the accomplishment of target agreements Preparing external evaluation, accreditation 2

49 QA in Practice: Responsibilities Tasks of a QA-officer Assignments (continued): Promote and accompany organisational learning and communication Encouraging faculty to conduct QA Setting up Quality Circles Benchmarking (external communication) Establishing cooperative relationships with other universities QA in Practice: Responsibilities Skills of an QA-officer Knowledge Philosophy behind QA-mechanisms (evaluation, accreditation, ) Instruments Empirical research methods Personality Communication skills, advise others, give guidance Cognition Discover linkages of local information needs (department level) and university missions 3

50 QA in Practice: Responsibilities Number of QA-activists in Germany: People 250 People who work regularly and on a full/half time position in the field of QA. There is still a growing demand; the QAprofession is slowly emerging QA in Practice: Responsibilities The German case QA-project set up by HRK in the 1990 s At university level: establishment of QA-agencies since the late 1990 s In charge of research and counseling Evaluation Networks: mutually conducted evaluation and peer review of study programmes Tasks of the QO s: facilitation of site-visits, reporting, etc. 4

51 QA in Practice: Responsibilities Several Universities offering postgraduate programmes in QA Different orientations: international focus, OMorientation, research methods Master programmes in evaluation and higher education management QA in Practice: Responsibilities Perspectives International Cooperation Mutual training programmes Recognition of degrees Elaboration of curricula with emphasis on local/regional conditions Mechanisms for distinguishing a profession 5