NEAA Report Final

Transcription

1 1 Report of the Panel appointed to undertake a review of the National Evaluation and Accreditation Agency (NEAA) of the Republic of Bulgaria for the purposes of renewal of full membership of the European Association for Quality Assurance in Higher Education (ENQA) 23 March, 2014

2 2 Executive Summary This is the report of the review of the National Evaluation and Accreditation Agency (NEAA) of the Republic of Bulgaria undertaken in February, 2014, for the purpose of determining whether NEAA meets the criteria for renewal of full membership of the European Association for Quality Assurance in Higher Education (ENQA). The report describes: the background and context for the review how the review was carried out the place of the Agency in Bulgaria s quality assurance structure for HE and the Agency s main functions the Panel s assessment of the engagement of the Agency with the ENQA membership provisions/european Standards and Guidelines and its conclusions the Panel s general observations and recommendations for development.

3 3 Glossary of terms AC Accreditation Council of NEAA AERES Evaluation Agency for Research and Higher Education AFDASRB Act for the development of academic staff in the Republic of Bulgaria APCL Accreditation of prior certificated learning APL Accreditation of prior learning ARACIS Romanian agency for quality assurance in higher education BSC Balanced scorecard CEENQA/CEEN Central and Eastern European Network of Quality Assurance Agencies in Higher Education CIPP Context, input, process, product CNE French National Evaluation Committee (AERES) ECTS European Credit Transfer and Accumulation System EHEA European Higher Education Area ENIC NARIC European Network of Information Centres National Academic Recognition Information Centres ENQA European Association for Quality Assurance in Higher Education EQAR European Quality Assurance Register for Higher Education ESG European Standards and Guidelines for Quality Assurance in the European Higher Education Area ESU European Students Union EUA European University Association FQ-EHEA The Framework for Qualifications of the European Higher Education Area HEA Higher Education Act HEI Higher Education Institution INQAAHE International Network for Quality Assurance Agencies in Higher Education IQA Internal Quality Assurance NASC National Assembly of Students' Councils NEAA National Evaluation and Accreditation Agency of the Republic of Bulgaria NEASC New England Association of Schools and Colleges R&D Research and development SC Standing Committees of NEAA SEE South-Eastern Europe SG Secretary General of NEAA SWOT Strengths, weaknesses, opportunities, threats NEAA website:

4 4 Background and outline of the review process 1. The Statutes of ENQA require all member agencies to undergo an external cyclical review, at least once every five years, in order to verify that they fulfil the membership provisions. In November 2004, the General Assembly of ENQA agreed that the third part of the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) should be incorporated into the membership provisions of its (then) Regulations (now Statutes). Substantial compliance with the ESG thus became the principal criterion for membership of ENQA. The ESG were subsequently adopted at the Bergen ministerial meeting of the Bologna Process in The third part of the ESG covers the approach to be taken to the cyclical external reviews of quality assurance and accreditation agencies. This external review of NEAA was conducted in line with the process described in the ESG (third edition) and in the accompanying ENQA Guidelines for external reviews of quality assurance agencies in the European Higher Education Area and in accordance with the timeline set out in the Terms of Reference determined for this particular review. Purpose and initiation of the review process 2. NEAA was last evaluated for the purpose of acquiring ENQA membership in July, 2008, with a positive decision taken by the ENQA Board. Therefore, in accordance with ENQA standard requirements, NEAA was due to undergo a new evaluation process for the purpose of extending its membership in ENQA, which also serves as an indicator for registration in EQAR. Following NEAA s application to ENQA to initiate this process, ENQA established an evaluation Panel, the composition of which met standard ENQA practice and which was duly briefed by ENQA prior to the Panel site visit to NEAA in Sofia, Bulgaria. Membership of the Panel 3. The members of the Panel appointed by ENQA to undertake the review were: Peter Findlay, (Chair of the Panel), Assistant Director, Reviews Group, Quality Assurance Agency for Higher Education (QAA), (UK) Paul Mitchell, (Secretary to the Panel), Director of Mega Mitchell Consulting Ltd., (UK) Oxana Matveeva, Deputy Head, Accreditation Office, National Centre for Public Accreditation (NCPA), (Russia) Jean-Marc Rapp, (EUA nomination), Professor at the University of Lausanne Law Faculty, Director of the Business Law Centre (CEDIDAC), (Switzerland) Liliya Ivanova, (ESU nomination), ESU Executive Committee member ( ), Master in International Economic Relations at the University of National and World Economy, (Bulgaria) Terms of Reference for the Review 4. ENQA has identified two types of external review which may be undertaken for the purpose of seeking membership: Type A - the sole purpose of which is to fulfil the periodic external review requirement for ENQA membership Type B - a review which has a number of purposes, only one of which is to fulfil the periodic external review requirement of ENQA membership.

5 5 This review was a Type A review, and evaluates how, and to what extent, NEAA fulfils the criteria for full ENQA membership and thus the Standards and Guidelines for Quality Assurance in the European Higher Education Area. The review was co-ordinated by ENQA itself at the invitation of NEAA. Approach and Procedure 5. The review was carried out using a process designed and managed by the Panel following established ENQA practice, independently of NEAA. The Panel sought to conduct the review in a manner that was not only professional and courteous, but also constructively searching and challenging. NEAA produced a self-evaluation report and the Panel also sought further supporting documentation following its initial analysis of the self-evaluation. The Panel conducted a site visit to explore further the content of the self-evaluation and to clarify points at issue. Finally, the review panel produced the present final report on the basis of the self-evaluation report, the site visit and its findings. As part of the process, the Panel has provided an opportunity for NEAA to comment on the factual accuracy of the draft report. In fulfilling the purposes of the review, the Panel has: considered the broad professional and political contexts within which NEAA operates considered a self-evaluation document prepared by NEAA, a range of supporting documents and web-based materials submitted in advance of the site visit (Appendix 2) considered additional documentation relevant to the Panel s lines of enquiry during the site visit (Appendix 2) conducted a two-day visit to NEAA (6 7 February 2014), at NEAA s headquarters in Sofia, Bulgaria (Appendix 1) met a range of stakeholders (from categories selected by the Panel) (Appendix 1), representative of all NEAA s operations, including: o President of the Council, Vice-President, immediate past-president and Secretary General o A sample of Chairmen of NEAA s Standing Committees o A sample of external experts who have acted in different NEAA procedures o A representative of the Ministry of Education o A sample of senior representatives from HEIs o A cross section of NEAA s professional administrative staff o NEAA staff who prepared the self-evaluation o Members of NEAA s Quality Assurance Standing Committee o A sample of undergraduate and postgraduate students with experience of NEAA s quality assurance procedures o Sample of stakeholders from civic society and employer/professional bodies. Self-evaluation document 6. The self-evaluation document submitted by NEAA (2013) comprised an account of the following principal areas: Foreword (by NEAA s President) National Evaluation and Accreditation Agency in the system of Bulgarian academic institutions

6 6 Description and evaluation of Agency activity in the light of quality standards in the European Higher Education Area (Compliance with European standards regarding external quality agencies (Part 3 of ESG) and with European standards regarding external quality assurance of higher education (Part 2 of ESG)) Appendices o Appendix A: Financial resources o Appendix B: Organisation chart o Appendix C: The Corrector-NEAA system o Appendix D: Abbreviations used in the report 7. The document had been circulated in draft to NEAA staff and signed off by Council members; there had been some stakeholder input in the later stages of preparation. As a tool for self improvement, the Panel considered the selfevaluation document to be weak. It focused primarily on NEAA s role in implementing the various laws and regulations in force underpinning an extremely intensive accreditation system, with very little reflection on NEAA s capacity for shaping, interpreting, influencing and learning from the application of these procedures and for delivering enhancement. The Panel therefore felt that the unique opportunity presented through undertaking a self-evaluation had not been fully taken up. The SER would have been strengthened had it contained greater evidence of a capacity for self-reflection and a more meaningful analysis of the organisation s future direction and strategy. The SER was not wellreferenced in relation to supporting documentation, either through appendices, through electronic media or web references. (The Agency s website was not fully up to date). In order to complete its task, the Panel had to seek additional basic documentation both before and during the site visit; this significantly slowed the Panel s programme of work.

7 7 Higher education system in the Republic of Bulgaria 8. The Panel heard that Bulgaria is a country in which there is a strong national commitment to higher education in all its forms and levels. The main purpose of the HE system is to develop highly qualified manpower for the benefit of Bulgarian society, as well as to foster the development of research and culture. In quantitative terms, the aim is that by 2020, 30% of employees should hold a degree and 90 % of them should have a job related to their field of study. 9. The State sets the broad conditions for access to the Bulgarian higher education system and sets the broad framework for its development. Government therefore: develops and implements national policy for the development of higher education and safeguards the academic autonomy of higher education ensures that appropriate quality of provision is maintained in both teaching and research subsidises the education of students in state universities creates and maintains a system for providing loans to students for the payment of tuition fees provides property to the state higher schools and provides tax and other incentives for all higher schools in support of their activities defines the conditions for state recognition of awards issued by the higher schools in Bulgaria and from abroad devolves delegated authority for quality and standards to the National Evaluation and Accreditation Agency 10. Bulgarian higher schools have adopted an educational model which maps to the reforms within the EHEA. At present, higher education in Bulgaria consists of three types of institution: Universities, which offer courses across a wide range of fields, such as humanities, environmental, public and technical programmes Specialized institutions of higher education Individual colleges. These three categories of institution can be found in both the public and private sectors. In the second half of 2013, the system of Bulgarian university education comprised 51 institutions as follows: 42 universities and higher schools (36 public and 6 private) 9 individual colleges (8 private and one state owned). Seven universities in Bulgaria operate with more than 15,000 students. In 7 education institutions, the capacity is for fewer than 1,500 students and in 28 education institutions it is below 5,000 students. More than 40% of higher schools and students studying in Bulgaria are concentrated in the capital city, Sofia. 11. Higher schools have been established in almost all the population centres of the country s six regions and 10 higher schools have established branch operations in other population centres outside their own immediate region. This model of widely distributed higher education institutions across the country accelerated further after the adoption of the Higher Education Act in The highly devolved model provides access to higher education in more remote and/or less-developed regions, where the standard of living of the population

8 8 seriously constrains mobility and access to the 7 established major universities of the country. The social function of some of the smaller higher schools and/or their affiliated units has been further enhanced in recent years by state investment in the development of distance learning in higher education. The Panel was told that in the case of the traditional institutions of higher education, the institutions largely operate at full capacity, although nationally there is a declining demographic amongst 18 year olds. 12. Overall analysis of the higher schools network in the country indicates some major trends. The South-western region has the most concentrated network of higher schools, where there are 22 higher schools (43 % of their total number in Bulgaria in the city of Sofia), followed by South-central region (with 7 higher schools in the city of Plovdiv), and North-eastern (with 5 higher schools in the city of Varna). Private higher schools operate in accordance with market forces, with each looking for a niche market for the training of specialists. Many of the higher schools share a relatively narrow specialization (for example business and law), which can lead both to inefficient use of resources (through duplication), but which can also stimulate constructive competition between higher education institutions. Establishment and development of NEAA 13. The Panel was informed that the purposes of external evaluation and accreditation undertaken by NEAA are twofold: firstly to promote higher education, to develop its potential and to improve and maintain the quality of education and to support quality enhancement Secondly to use the outputs from accreditation in the formation of Government policy towards the higher schools. 14. The legal framework underpinning Bulgarian higher education was revised with the adoption of the Higher Education Act (HEA) in This came six years after the introduction of major democratic and social change in Bulgaria. This law, determined the overall functions, structure and responsibilities of NEAA, and required the Agency to develop overall principles for evaluation and accreditation and to devise specific criteria and procedures both for institutional and programme accreditation and also for assessing proposals for the opening and/or re-structuring of higher schools. 15. Following the further amendment of the HEA with effect from April, 2004, NEAA was tasked to perform the functions of a specialized agency for the evaluation, accreditation and quality control of the activities of the higher schools, together with responsibility for system-wide monitoring and control the latter constituting a new activity for NEAA. 16. The HEA thus provides the broad legal framework for the implementation of procedures for the evaluation of projects, for opening new professional fields and for programme accreditation; a special order was introduced for the implementation of procedures for evaluating proposals and programmes from the regulated professions. Following further amendments to the HEA in August 2011, NEAA was given a stronger legal framework with which to regulate the quality of higher education through the granting of the right to sanction higher education institutions failing to implement recommendations from institutional and programme accreditation. In 2012, the Statutes of NEAA were further amended and supplemented to reflect developments within the EHEA.

9 9 17. In 2010, the Act for the Development of Academic Staff in the Republic of Bulgaria (AFDASRB) came into force. This law is the principal driver for regulating the career development of university teachers and scientists across the country as a whole. NEAA has no powers directly conferred by the AFDASRB, but there are indirect functions, insofar as analysis of institutional activity in support of the development of academic staff features prominently in NEAA s accreditation criteria. This places NEAA indirectly in the position of a monitoring body for the quality of academic staff development across higher education and also across the units of the Bulgarian Academy of Sciences, Agricultural Academy and the other entities of scientific activity in Bulgaria. Priorities and activities of the Agency 18. NEAA has established its own strategic priorities and objectives, reflecting national needs; these are expressed in its strategic plan for NEAA states that a key strand is national articulation with European higher education policy, and particular attention is being paid to the harmonisation of the activities of NEAA with the requirements of the Bologna process and the alignment of the Bulgarian higher education system with the framework of EHEA. 19. NEAA currently carries out the following types of procedure: Procedures for institutional accreditation Procedures for programme accreditation of programmes from the regulated professions Procedures for programme accreditation of programmes outside the regulated professions Procedures for programme accreditation of courses delivered by distance learning. Procedures for the evaluation and accreditation of PhD programmes. Procedures for the evaluation of proposals for the opening or reorganisation of a higher school Procedures for post-accreditation monitoring 20. The table below summarises the activities carried out by the Agency in the period External quality assurance * procedure Institutional accreditation Programme accreditation of professional fields Programme accreditation of degree programmes in regulated professions Programme accreditation of doctoral degree programmes Total Evaluation of new projects for restructuring of higher schools Evaluation of new projects for opening and restructuring of basic units and branches of higher schools Evaluation of new projects for opening a degree programme in regulated professions Evaluation of new projects for opening professional fields

10 10 Total Institutional accreditation Internal quality assurance systems of HEIs Post-accreditation Programme monitoring of quality accreditation of professional fields New project Total Procedures for change of educational capacity *As at September It is difficult for the Agency to regulate the flow of business coming to it, in terms of the volume of incoming requests. Much of its activity is triggered by the legacy of timescales from previous accreditations and by decisions made by individual HEIs in relation to proposed changes to the scale or range of their activities, including decisions to offer new or revised programmes. The total number of procedures for institutional and programme accreditation in the period 2009 to 2013 has varied annually. The busiest year in the period was 2010, with a total number of 578 procedures; 2012 and 2013 were less busy in number of individual procedures, reflecting commencement of the new cycle of the procedures for institutional accreditations. Management of this uneven pattern of demand has been very challenging for NEAA. 21. NEAA has made a series of recent changes in its work model and in systems connected with the assessment and accreditation of institutions and programmes. These include: New criteria for the evaluation of proposals for the opening and reorganisation of higher schools and basic units and for opening professional programmes from the regulated professions; new criteria for programme accreditation of training in the educational and scientific degree of "Doctor"; and new criteria for distance learning Voting rules under the procedures for evaluation and accreditation have been adopted by the Council, as well as internal rules for imposing sanctions on higher schools as required by the HEA Revision of the criteria system for evaluation and for institutional and programme accreditation Increased focus has been placed on the evaluation of research activities, (also reflecting a concern expressed to the Panel by some research-intensive universities). 22. Following completion of a first full round of national evaluations, a new system for the evaluation and accreditation of higher education was introduced in June 2011, which includes a more detailed grading structure, intended to differentiate more sharply between the various higher schools. At the time of the review, 70% of programmes had been granted accreditation under the new system. Four areas of activity in the higher school are subject to evaluation. They are: educational activities, research, management, and competitiveness of

11 11 the higher school. The new grading system is based on a scale from one to ten. The aggregated final assessment is the product of a number of measured indicators, each of which has its specific value. 23. NEAA operates a complex system for determining periods of accreditation, at both institutional and programme level. For example, following consideration of a report on institutional or programme accreditation, the Agency determines the period of accreditation, according to a scale ranging from 0 to 10. The period of validity of the accreditation varies, depending on the assessment, as follows: six years with an evaluation grade of 9.00 to 10; five years with an evaluation grade of 7.00 to 8.99; four years with an evaluation grade of 5.00 to 6.99; three years with an evaluation grade of 4.00 to In the case of an assessment ranging from 0 to 3.99, accreditation is not awarded. Revocation of institutional accreditation can be made when a higher school has received an assessment from 0 to 3.99 in one or several of the following criteria: Presence and status of an internal quality assurance system Profile and qualifications of academic staff Logistic provision of the training process. 24. Similar calibrated gradings apply to other areas of evaluation activity. For example, at subject level, an evaluation grade of less than 4.00 in one or more of the following criteria will prompt a negative assessment for programme accreditation: Training documentation and the educational process Profile and qualifications of the academic staff Logistical and information base, customized for training Higher schools with an assessment in programme accreditation from 4.00 to 4.99 have the right to teach only to the level of "Bachelor" in the relevant field or specialty from the regulated professions. When awarded an evaluation grade from 5.00 to at programme accreditation, higher schools have the right to teach to the levels of "Bachelor" and "Master". Higher schools and organizations may conduct training for the degree of PhD if they have received an evaluation grade from 8.00 to Proposals for the opening of higher schools, as well as for new faculties, affiliates, colleges and professional fields and specializations from the regulated professions are assigned an overall positive or negative assessment. When the assessment is positive, the validity of accreditation is for a period of 2 to 3 years, after which follows a regular accreditation; in the case of a negative assessment, revocation of accreditation is made. In all cases of accreditation revocation the Agency formulates clear reasons, based on clear criteria. 26. Higher schools, having received accreditation, may carry out training of undergraduate and postgraduate students under agreements for joint training activities with foreign universities; the foreign higher schools must meet the following two conditions: Be accredited by an agency which is a member of the European Association for Quality Assurance in higher education (ENQA) and/or entered in the European quality assurance register for higher education (EQAR)

12 12 Have accreditation under the relevant national legislation in order to participate in transnational collaboration NEAA Structure 27. The overall functions and structure of NEAA are set in Articles 86 and 87 of the HEA. The governing authorities of the Agency are its Accreditation Council and its President (Chairman). The Accreditation Council consists of a Chairman and 10 members professors and associate professors from areas of higher education, one of whom is also the vice president responsible for postaccreditation monitoring and control. The Council comprises: Six representatives of higher schools One representative of the Bulgarian Academy of Sciences and one representative of the Agricultural Academy Two representatives of the Ministry of Education and Science. The President of NEAA represents the agency and oversees its activities. He/she is appointed by the Prime Minister on a full-time basis, on the nomination of the Minister of Education and Science. NEAA Standing Committees 28. The Accreditation Council delegates much of its detailed work to nine Standing Committees - eight covering specific subject areas of higher education and one covering overall post-accreditation monitoring and control. The committees each consist of 3 to 7 members, one of whom acts as the Chairman. Members of the committees are appointed for a 3-year term of office by the President of the Accreditation Council. Appointments are renewable for a second term (only). The current Standing Committees are: Pedagogical Sciences Humanities and Arts Social and Legal Sciences and National Security Economic Sciences and Management Natural Sciences, Mathematics and Computer Science Technical Sciences and Military Science Agricultural Sciences and Veterinary Medicine Healthcare and Sport Post-accreditation Monitoring and Control. 29. The Chairs of Standing Committees are appointed on a full-time basis, with their members on part-time contracts. The Standing Committees oversee the procedures for: programme accreditation of professional fields; programme accreditation of scientific disciplines outside those of the regulated professions; and evaluation of projects for opening new professional fields. The Standing Committees submit for approval to the President of the Accreditation Council the composition and tasks of the expert groups performing procedures for institutional and programme accreditation. They monitor the work of the expert groups and appoint a specific member of the Standing Committee to monitor each procedure. 30. The Standing Committee on Post-Accreditation Monitoring and Control: Submits for approval by the Accreditation Council individual timetables for post-accreditation monitoring and control

13 13 Submits for approval by the Accreditation Council the composition and tasks of the expert groups, in accordance with the nature and content of the procedure for post-accreditation monitoring and control Oversees the work of the expert groups and on the basis of their reports draws up protocols for the checks carried out Prepares and after consultation with the Vice President, presents to the Accreditation Council, reports on the results of post-accreditation monitoring and control Organizes and conducts the training of experts in the field of Postaccreditation monitoring and control 31. The Accreditation Council, on the proposal of the relevant Standing Committee approves the composition and terms of reference for short-life expert groups. The members of the expert group make on site visits to the higher education institutions under evaluation and draw up a report to the appropriate Standing Committee. The expert group comprises academic peers together with undergraduate or postgraduate students; it can sometimes include an international expert. In cases of institutional failure to implement the required improvements within the specified period or on a proposal from the Minister of Education and Science for withdrawal of accreditation, the Standing Committee may make a recommendation to the Accreditation Council to withdraw the relevant institutional or programme accreditation. The Panel noted that the provision for direct reference by the Minister for Education and Science had never been invoked. Management Structure of NEAA 32. The management structure is set out in diagrammatic form below: PRESIDENT VICE PRESIDENT ACCREDITATION COUNCIL SECRETARY GENERAL General Administration Legal Counsel Accounting Specialised Administration Chairmen of Standing Committees Experts to Standing Committees

14 14 The general and specialised underpinning administration comprises ten full-time officers and eight part-time staff. External contacts 33. NEAA maintains effective working links in developing and cascading policy with: the Bulgarian Ministry of Education and Science, the Bulgarian Council of Rectors, national representatives of Student Councils and of other student organizations. Officers of the Bulgarian Academy of Sciences, of the Agricultural Academy and other research organisations. Update on 2008 Review 34. In its 2008 report on NEAA, the ENQA review team highlighted a number of areas for development. Below is a chart setting out the Panel s assessment of progress made in key areas since 2008: 2008 Report: areas for improvement NEAA follow up (as at January, 2014) 1 NEAA would benefit from defining annual action plans. These plans should include, apart from a description on the different Achieved. Annual planning has been introduced. The Panel has reviewed NEAA s evaluation and accreditation activities of the current annual plans. It Agency, references on the conditions considers these to be required for Bulgarian higher education and adequate for current research to achieve full integration into the purposes, but with scope for European context. further development and sophistication, designed to capture external factors 2 A more strategic definition and management of the budget, specifying the costs of the different accreditation activities, would allow NEAA to take on a more strategic role in the evaluation of Bulgarian higher education. Fees paid to the external experts need to be reviewed, especially to give an incentive to international experts from abroad. 3 It may also be important for the future of NEAA to seek a better gender balance and to involve a greater proportion of persons impacting on NEAA s work In progress. Financial analyses presented to the Panel reflecting programme based activity, demonstrate an increased awareness of costs. The Agency is under considerable financial pressure, driven by the volume (and to some extent the complexity) of the accreditation processes which it undertakes. Further progress needs to be made on funding provision for the involvement of international experts. In progress. An improved gender balance has been achieved (e.g. total Standing

15 15 from the next generation and from more diverse backgrounds (including professionals) in decision-making bodies. 4 Students participating in expert teams should have the status of full members (not just that of external advisers to the expert team reporting on a limited number of aspects). The same applies with respect to foreign experts and professionals. Special attention must be paid to include the latter. Committee membership now comprises 30 men and 22 women). The full time administration is entirely female. Age structure however remains significantly skewed towards the age ranges. Membership of the Agency s Council remains confined to members of the Professoriate and could with benefit be further reviewed. Increased involvement of employers and civic society in both its deliberations and activities has yet to be fully achieved. Partly Achieved. Since the last report, an amendment has been made to accord full status to student team members. The Panel heard however of some instances where Standing Committees had opted to deploy student members in a slightly different way. The Panel urges NEAA to issue to its Standing Committees unequivocal, standardised requirements on this topic for the avoidance of doubt. Foreign experts and other constituencies (e.g. business, professional bodies and employers) are not yet fully integrated into NEAA s procedures and activities (see discussion under Standard 2.4). 5 The process of drawing up the accreditation reports could be made less burdensome: the reports drawn up by the expert team, the student, the international expert, the Standing Committee and the Accreditation Council should be consolidated in a single procedure and document. This would also facilitate their publication. Achieved. All inputs are now consolidated in a single procedure and document.

16 16 6 NEAA should take a closer look at the requirement to publish all reports and should find a way to put more information at the disposal of users and stakeholders. 7 The training of external experts should be improved. More efforts should be made to train staff and experts about the EHEA and to expose them to European/international counterparts. 8 The system of awarding accreditation grades very good, good, satisfactory - offers interesting possibilities but needs to be used with caution (less generously) if it is to keep (or to restore) its credibility. It should be reviewed in order to generate more appropriate differentiation and to enhance its international readability and credibility. 9 NEAA should also pay closer attention to its role in providing system-wide analyses for the overall improvement of Bulgarian higher education and its optimal insertion in the EHEA. 10 A related need is to check whether all the various external quality assurance procedures that the Agency develops are really essential for the enhancement of the quality in Bulgarian Higher Education Programme and institutional accreditation could be merged in the same process and pilot procedures could be launched in order to fine-tune new procedures before they are generalised. 11 NEAA would benefit from inviting HEIs, students and professional organizations to join in during the design stage of external quality assurance procedures. 12 Public visibility and recognition of the Agency should be enhanced. Wider debates on quality assurance issues should be generated by NEAA in order to achieve better understanding on its role in the HE spectrum and to raise the level of ownership of the stakeholders over its actions. Not achieved. See Para. 56 Achieved. Since the last report, NEAA has enhanced the quality of its training. New Reviewers are now thoroughly inducted in the process and periodic training and updating sessions are arranged. New reviewers are assigned a mentor. Achieved. NEAA has now adopted a more finely differentiated grading scheme. The cost/benefit analysis of this fine-grained level of scrutiny has yet to be fully assessed. In progress. See Para. 60 In progress See Paras (and recommendation) In progress University managers interviewed by the Panel confirmed their increased institutional involvement in the planning and roll out of QA procedures In progress See Para. 52

17 The Panel s overall assessment is that NEAA has responded appropriately to many of the recommendations contained in the 2008 ENQA report, but that further work still needs to be done in some areas to carry these through to satisfactory completion.

18 18 NEAA compliance with the Standards and Guidelines for Quality Assurance in the European Higher Education Area ESG Part 2: European standards and guidelines for the external quality assurance of higher education ESG 2.1 Use of internal quality assurance procedures (ENQA Criterion 1) Standard: External quality assurance procedures should take into account the effectiveness of the internal quality assurance processes described in Part 1 of the European Standards and Guidelines. Guidelines: The standards for internal quality assurance contained in Part 1 provide a valuable basis for the external quality assessment process. It is important that the institutions own internal policies and procedures are carefully evaluated in the course of external procedures, to determine the extent to which the standards are being met. If higher education institutions are to be able to demonstrate the effectiveness of their own internal quality assurance processes, and if those processes properly assure quality and standards, then external processes might be less intensive than otherwise. NEAA compliance 36. The HEA requires every higher school in Bulgaria to develop and implement an internal system for quality assurance and management, including provision for student feedback. NEAA procedures include criteria and indicators for the evaluation of established systems for monitoring, evaluation, maintenance and improvement of quality. NEAA s procedures for institutional post-accreditation monitoring and control similarly assess the development and maintenance of internal quality assurance systems in the higher education institutions. Following the successful completion by NEAA of two major cycles of institutional and programme accreditation, it has confirmed that each higher school has in place a functioning internal quality assurance system. 37. The Panel has confirmed that the starting point for NEAA s institutional review processes is based firmly on the institutions' self-evaluations of their own processes. The Panel has examined and explored NEAA's institutional review processes, mapped against the ESG, and confirms that these are effective in testing and challenging institutions' internal quality assurance policies and their procedures for managing quality and standards. The Panel also noted 17 HE institutions in Bulgaria have achieved the award of quality certificate ISO 9001, a helpful additional indicator of the health of the relevant institutions management systems.

19 The Panel therefore confirms that the external quality assurance procedures that NEAA uses, take into account fully the effectiveness of the internal processes described in Part 1 of the ESG. Panel judgement: Fully compliant. Panel Recommendations None.

20 20 ESG 2.2 Development of external quality assurance processes (ENQA Criterion 1 cont.) Standard: The aims and objectives of quality assurance processes should be determined before the processes themselves are developed, by all those responsible (including higher education institutions) and should be published with a description of the procedures to be used. Guidelines: In order to ensure clarity of purpose and transparency of procedures, external quality assurance methods should be designed and developed through a process involving key stakeholders, including higher education institutions. The procedures that are finally agreed should be published and should contain explicit statements of the aims and objectives of the processes as well as a description of the procedures to be used. As external quality assurance makes demands on the institutions involved, a preliminary impact assessment should be undertaken to ensure that the procedures to be adopted are appropriate and do not interfere more than necessary with the normal work of higher education institutions. NEAA compliance 39. The procedures for external quality assurance developed by NEAA are publicly available on the Agency's website. Whenever replacement or a significant revision of a procedure is proposed, this will be first discussed with the higher education institutions, relevant sector organizations, experts in the field and the Ministry. Any significant adjustments to the criteria and procedures are normally discussed with the Council of Rectors, but the final decision is nevertheless taken by NEAA. 40. The different types of procedures for evaluation and accreditation, developed and applied by NEAA include clear descriptions of the different activities associated with each procedure and of the various participants involved at each stage. The documents are publicly available on the Agency s website. Whenever changes are formally agreed, users are notified - by letter and copies of the relevant documents, so they can be aware of any changes. Officers of the Agency are expected to act in an advisory role to staff in higher education institutions, regarding all aspects of external evaluation and accreditation. Panel judgement: Fully compliant. Panel Recommendations None.

21 21 NEAA compliance NEAA publishes a full suite of criteria and procedures which underpin its ESG 2.3 Criteria for decisions (ENQA Criterion 1 cont.) Standard: Any formal decisions made as a result of an external quality assurance activity should be based on explicit published criteria that are applied consistently. Guidelines: Formal decisions made by quality assurance agencies have a significant impact on the institutions and programmes that are judged. In the interests of equity and reliability, decisions should be based on published criteria and interpreted in a consistent manner. Conclusions should be based on recorded evidence and agencies should have in place ways of moderating conclusions, if necessary. NEAA compliance 41. Evaluation, accreditation and monitoring activities are carried out in accordance with uniform criteria, which are published on the Agency's website as follows: Criteria for institutional accreditation of higher education institutions. Criteria for programme accreditation of a professional field and a major from the regulated professions. Criteria for programme accreditation of the training in educational and scientific degree of "Doctor". Criteria for evaluation of projects for opening and transformation of a higher school. Criteria for evaluation of projects for opening and transformation of basic units and/or an affiliate of a higher school. Criteria for evaluation of projects for opening of a professional field and a major from regulated professions. Criteria for evaluation of specializations in distance learning. Criteria for amendment to capacity. Criteria for post-accreditation monitoring Procedures for post-accreditation monitoring 42. Consistency in the interpretation and application of the criteria is achieved through: The training of expert groups, Professional oversight of expert groups by individual nominated Council members and by trained NEAA staff Regular dialogue between the Accreditation Council and the Standing Committees, with periodic consideration of specific case studies, Availability of clear guidance and rules, according to which the procedures for assessment and accreditation are carried out For expert teams, provision of methodological guidelines for the preparation of a report by the expert group For Standing Committees, provision of:

22 22 o o Rules for the conduct of voting procedures by the standing committees in areas of higher education for programme accreditation outside those from the regulated professions. Rules for the conduct of programme accreditation of professional fields. o Methodological explanations for the determination of assessments for institutional and programme accreditation, for the evaluation of projects for opening or transformation of higher schools, basic units and professional fields and majors from the regulated professions. o Methodological guidelines for the preparation of a report by the Standing Commission to the Accreditation Council (for institutional accreditation). For Accreditation Council, provision of: o Rules for the conduct of a voting procedure by the Accreditation Council for institutional accreditation of higher education institutions. o Rules for the conduct of voting procedures by the Accreditation Council in the evaluation of projects for the opening and transformation of a higher education institution, basic units and affiliates of a higher school and majors from the regulated professions. Chairmen of Standing Committees have a key moderating role across their specialist area. Institutions also have the right to comment on any factual inaccuracy and other discrepancies in the report. This opinion is submitted to the Agency before any final decision is taken. 43. NEAA s procedures and criteria are well-documented. NEAA s Council maintains an effective overview of its operations and has arrangements in place to ensure consistency. Panel judgement: Fully compliant. Panel Recommendations None.

23 23 ESG 2.4 Processes fit for purpose (ENQA Criterion 1 cont.) Standard: All external quality assurance processes should be designed specifically to ensure their fitness to achieve the aims and objectives set for them. Guidelines: Quality assurance agencies within the EHEA undertake different external processes for different purposes and in different ways. It is of the first importance that agencies should operate procedures which are fit for their own defined and published purposes. Experience has shown, however, that there are some widely-used elements of external review processes which not only help to ensure their validity, reliability and usefulness, but also provide a basis for the European dimension to quality assurance. Amongst these elements the following are particularly noteworthy: insistence that the experts undertaking the external quality assurance activity have appropriate skills and are competent to perform their task the exercise of care in the selection of experts the provision of appropriate briefing or training for experts the use of international experts participation of students ensuring that the review procedures used are sufficient to provide adequate evidence to support the findings and conclusions reached the use of the self-evaluation/site visit/draft report/published report/follow-up model of review recognition of the importance of institutional improvement and enhancement policies as a fundamental element in the assurance of quality. NEAA compliance 44. The basic rules for the formation of expert groups are set out in the Statute regulating the activities of NEAA. Expert groups have temporary status and their composition is approved by the Accreditation Council on the nomination of the Standing Committees and by the President of the Council. Special attention is paid to the need for a good balance within teams, to include members who have professional experience from, and professional commitment to different higher education institutions and scientific organizations. 45. In the Statute for the activity of NEAA (art. 13, paragraph 3), experts can be Bulgarian and/or foreign individuals, including representatives of professional organizations and associations of employers, as well as students and PhD students. International experts are invited normally when the institutions undergoing accreditation, request them. The desire of the Accreditation Council to include more international experts in its work presents some challenges attributed to the relatively low fees which are paid and which do not attract many qualified European experts. 46. Experts appointed to carry out quality assurance activities for NEAA are selected according to the following specific criteria: Professional experience in higher education Credibility of the expert in the academic community as a teacher and as a researcher Managerial experience in higher education

24 24 Experience in the evaluation of higher education institutions, including outside NEAA (scientific projects and programmes, educational initiatives, etc.) Knowledge of the legal basis regulating an HEI s activities and academic work Standing of the institution in which the expert works or has worked Ability to work as part of a team and to exercise impartiality, objectivity, confidentiality, constructiveness and a business-like approach to work Absence of any conflict of interest with the institution under review Ability to integrate with the activities of the expert group Compliance with the requirements of art. 13, paragraph 3 of NEAA Statutes Confirmed willingness and consent for participation in the procedure 47. NEAA involves students and doctoral students in the work of expert teams. It has developed special "Rules for the participation of students and PhD students in procedures for evaluation and accreditation". The selection of student experts is carried out in partnership with the national student council organisations, the managements of higher schools and on the basis of established personal contacts of the expert groups with representatives of the students during assessment visits. Students are now fully official members of expert groups. There remains, however, some unevenness in the involvement of students in all aspects of the experts' work. In particular, it was noted that students did not have the same level of participation in training events, and student membership of the Standing Committees was not consistent. 48. The training of experts is considered to be an essential part of the Agency's work. NEAA appoints a member from the relevant standing committee, who is tasked with the function of a mentor. He/she normally instructs experts together with the Chairman of the Standing Committee. Besides explaining their duties and providing the necessary information, special attention is paid to the following aspects during briefing and preparation for an onsite visit to an institution: Advance familiarisation with the documentation provided by the higher education institution Clarification of their role and powers as an inspection body and understanding the parameters Focusing the experts' attention on the need to verify that recommendations from any previous accreditation have been implemented Drawing the attention of the experts to their obligation to examine and evaluate the effectiveness of the internal quality assurance system in the higher education institution or scientific organisation concerned Clarification of all indicators and characteristics of all criteria, including the reliability of numerical indicators and parameters Emphasis on meetings with the heads of teaching unit(s), students and PhD students Preparing for meetings with employers and other stakeholders. 49. Workshops are held for experts assessing the same discipline or professional field. This type of event provides an opportunity for experts to share their experiences and to comment on the overall development of a professional field. At these workshops they harmonise their approaches in the application of the criteria and procedures and discuss specific case studies.