Assessment of the Polish evaluation system

Transcription

1 Daniela Breazu, Steluța Bulăceanu, Anca Simion, Cătălin Costache, Alicja Weremiuk Assessment of the Polish evaluation system Peer Review Report Warsaw

2 List of contents Acknowledgements...3 List of abbreviations...4 Introduction - context of the study...5 Methodological approach and its limitations...8 Polish evaluation system description...10 Hypothesis of the study...12 Verification of the hypothesis...14 Evaluation planning...14 Preparation of ToR and commissioning of evaluations...16 Evaluation management...17 Follow-up process...20 Summary - SWOT analysis of the Polish evaluation system...25 Recommendations...26 Improvements within the Polish evaluation system...26 Applications for the Romanian evaluation system...29 Appendix 1 Romanian best practices...33 Appendix 2 IDI questionnaire...34 National Evaluation Unit questionnaire...34 The MA and IB Evaluation Units questionnaire...36 Appendix 3 - List of respondents

3 Acknowledgements (alphabetical order) Hanna Bałos, Stanisław Bienias, Renata Burzyńska-Wolny, Marta Bystrowska, Michał Czarnecki, Gabriela Duell, Marcin Fronczak, Agnieszka Gintowt-Dziewałtowska, Agnieszka Haber, Przemysław Herman, Jakub Jąkalski, Maciej Kościelski, Tomasz Kula, Małgorzata Lublińska, Cristina Macarie, Filip Maciągowski, Michał Macura, Claudia Magdalina, Agata Michałowska, Elżbieta Opałka, Jarosław Orliński, Despina Pascal, Barbara Porębska-Maciołek, Magdalena Ryznar, Ioana Stan, Piotr Strzębowszewski, Michał Sułkowski, Maciej Szałaj, Justyna Wenglorz, Grzegorz Witkowski, Tomasz Złakowski 3

4 List of abbreviations ACIS - Authority for Coordination of Structural Instruments (in Romania) AEP Annual Evaluation Plans CSF Common Strategic Framework ECU Evaluation Central Unit (in Romania) EU Evaluation Unit EWG Evaluation Working Group (in Romania) HC OP Human Capital Operational Programme (in Poland) IB Intermediate Body IDI Individual In-Depth Interview IE OP Innovative Economy Operational Programme (in Poland) I&E OP Infrastructure and Environment Operational Programme (in Poland) FGI Focus Group Interview MA Managing Authority MAEP - Multi-Annual Evaluation Plans NDP National Development Plan NEU the National Evaluation Unit (in Poland) NSRF - National Strategic Reference Framework OP Operational Programme ROP Regional Operational Programme SF Structural Funds SG Steering Groups SI Structural Instruments SOP Sectoral Operational Programme ToR Terms of Reference 4

5 Introduction - context of the study The internship of Romanian representatives in the Polish Ministry for Regional Development was financed from the project Developing the evaluation capacity of the evaluation units within Managing Authorities and ACIS. It took place between 18th and 29 th June During the internship four Romanian administration representatives were hosted in different departments of the Ministry. Trainee Function in Romania Polish MRD Department Time Mrs. Breazu Mr. Costache Daniela Cătălin Expert, Managing Authority for Sectoral Operational Programme (SOP) Transport Head of Unit, Managing Authority for SOP Transport Mrs. Anca Simion Counsellor, European Territorial Cooperation Directorate, Ministry of Regional Development and Tourism Public-Private Partnership Unit, Department of Individual Projects Preparation Programming and Evaluation Unit, Department of Infrastructural Projects Coordination Evaluation Unit, Department of Structural Policy Coordination Physical Progress Monitoring Unit, Department for Coordination of Implementation of the EU Funds Mrs. Bulăceanu Steluta Expert, Managing Authority for OP Administrative Capacity Development Evaluation Unit, Department for Management of European Social Fund Logic of the project The idea to introduce a new approach for organising this internship was agreed during the working meeting between both of the undersigned, which took place in the margins of an international conference held in Bucharest in April We agreed that the strategic result of the mission was to increase the evaluation capacity of the Managing Authorities which delegated their representatives for the internship. To achieve this result, the Romanian administration representatives who took part in the internship had to: learn in detail about the evaluation activities implemented in the Polish evaluation system, understand the strengths and weaknesses of different provisions as well as the context of their implementation, analyse the possibilities to use similar solutions in the Romanian evaluation system, tailored to the specific Romanian context. These, let us say, direct and immediate results would need to be triggered directly by activities undertaken during the mission. The challenge was to prepare the overall concept for carrying out such activities. Many studies and experiences confirm that one of the most effective approaches for building the evaluation capacity within the framework of Cohesion Policy evaluation is learning by 5

6 doing. We decided to put this approach into practice by organising a small evaluation carried out by Romanian representatives acting as peer reviewers. This would allow them to learn about the Polish evaluation system and prepare recommendations as to: what evaluation arrangements can be transferred from the Polish system and successfully integrated into Romanian evaluation practice; and what changes could be made to improve the Polish system. We agreed that the main product of the internship would be the Peer Review Report assessing the Polish evaluation system. Based on these assumptions we constructed our own logic chain for the project based on the theory of change. This logic chain is presented graphically in the diagram below: INPUT NEED Increased evaluation capacity in the Managing Authorities designating interns Elaboration of the internship concept - 4 interns from Romanian MA (38 man-days) - Polish Evaluation expert (11.5 man-days) - time resources of the Polish Ministry employees (8 man-days) Preparation of the evaluation OUTPUT - Peer Review Report (1) IMMEDIATE DIRECT RESULT Increase of participants knowledge on Polish evaluation arrangements, their strengths and weaknesses and context of their implementation Impact LONG-TERM STRATEGIC RESULT Increased evaluation capacity in the Managing Authorities designating interns OTHER FACTORS What do we want to change? The preparation of the Report required additional day-to-day coordination by a Polish-speaking expert. The expert chosen was familiar with the organisation of the Polish system and workings of the Ministry for Regional Development and experienced in preparing evaluation reports. Note for readers When reading this document, the following should be taken into account: The Romanian interns faced significant challenges: - limited time resources; - limited (or even lack of) experience in the role of acting evaluators; - lack of experience in the field of Cohesion Policy evaluation in general. Preparation of the perfect report was not the objective of this mission, the goal was to increase the evaluation capacity of those who prepared the report and the potential of the institutions who designated them! 6

7 Report structure The report consists of the following elements: methodology and its limitations - brief description, Polish evaluation system description, hypothesis of the study, verification of the hypothesis, SWOT analysis of Polish evaluation system as a summary, recommendations for Romanian and Polish evaluation systems. Stanislaw Bienias Claudia Magdalina Head of National Evaluation Unit Department for Structural Policy Coordination Ministry for Regional Development POLAND Head of Unit, Evaluation Central Unit Authority for Coordination of Structural Instruments Ministry of European Affairs ROMANIA 7

8 Methodological approach and its limitations As stated above, the time and human resources engaged within this project were limited. The methodology chosen was therefore simple. However, it embraced the full range of evaluation research phases (observation, analysis and assessment) as well as different types of information sources. Data gathering was based mainly on desk research which focused on the following documents: Guidelines concerning the Evaluation of Operational Programmes for , NEU, Warsaw 2007, National Strategic Reference Framework Evaluation Plan, NEU, Warsaw 2007, NSRF Evaluation System organization of the evaluation process of the NSRF, NEU, Warsaw 2007, Process of evaluation of the Cohesion Policy in Poland, Warsaw 2011, Evaluation step by step. The MA s recommendations for evaluations under HC OP. Update, HC OP MA, Warsaw 2011 Presentations were subsequently delivered on: Polish evaluation system (the NEU perspective), HC OP and I&E OP evaluation units and performance (the MAs perspective), system of managing evaluation findings and recommendations (the NEU perspective), practical application of the evaluation process (commissioners and evaluators perspective), evaluation capacity building initiatives (IB and Polish Evaluation Society perspective). In order also to collect relevant opinions, Individual In-depth Interviews (IDI) were conducted as follows: 1 with the NEU representative, 4 with sectoral OP evaluation units (MA level) representatives, 2 with sectoral OP representatives (MA level) responsible for programming and monitoring (IDI focus was on evaluation recommendations implementation), 1 with regional OP evaluation unit (MA level) representative (via telephone), 2 with regional OP evaluation units (IB level) representatives (in written, via ). To broaden the perspective from individual opinions, a Focus Group Interview (FGI) was conducted. 4 representatives from the NEU and sectoral OP evaluation units attended the FGI as well as 4 Romanian interns. The FGI was moderated by the external evaluator. The secondary and primary data collected was analysed within a SWOT framework. However, the following main methodological and organisational constraints should be highlighted: only two weeks were available to gather and assess the data, data was gathered mainly from central institutions (as the internship was organized in Warsaw), very few (3) regional evaluation units were involved. 8

9 It can be said nevertheless that triangulation was achieved - as far as this was possible given the time constraints. 9

10 Polish evaluation system description Fact sheet: - National Evaluation Unit, 24 evaluation units at MA level, 29 at primary IB level, 2 at secondary IB level evaluations % on-going evaluations - mainly external evaluations - around 60 active companies within the evaluation market Evaluation in Poland is carried out generally at two levels: CSF/NDP and OP. The National Evaluation Unit is responsible for the NDP/CSF and horizontal issues evaluations as well as the ex-ante evaluation of the National Strategic Reference Framework (NSRF) for The OP Evaluation Units, established in the MAs are responsible for carrying out thematic and smallscale evaluations mainly aimed at improving the quality of implementation mechanisms. Evaluation Units are located within each OP management structure. They are responsible for organisation of the evaluation process within their respective OP and at their level of implementation. Following the decentralization of the implementation system, the evaluation process for is also conducted at regional level. Sixteen regional Evaluation Units, one in each Voivodeship, handle tasks related to the evaluation of regional OPs, as well as carrying out research for the national level on the regional dimension. Each OP MA can transfer competences in evaluation to lower levels of implementation. As we move into the evaluation process, the key role of the National Evaluation Unit (NEU) must be acknowledged. The NEU is located within the Department of Structural Policy Coordination in the Ministry of Regional Development and is responsible for coordination of the whole system (work of other Evaluation Units). This entity has prepared a document called Guidelines on evaluation of Operational Programmes , together with the OP Managing Authorities. Its purpose is to: ensure compliance of OP evaluation with EU regulations, 10

11 define general competence areas involved in the process of NSRF and OP evaluation in , define the scope of evaluations in , describe the general principles of the evaluation process. The NEU cooperates at different levels with the OP Evaluation Units to coordinate implementation of the evaluation process. In order to prevent duplication and to ensure spread of good practice a Steering Group for CSF/NDP Evaluation has been established. In addition to co-ordinating evaluation efforts and facilitating information flows between stakeholders, the Steering Group approves the NDP/CSF final evaluation reports and ensures the application of standards across all Cohesion Policy evaluation activity. The Evaluation Steering Group at NSRF level is responsible for setting the directions for the work of all units. There are consultations with directors from other Evaluation Units, as well as heads of Evaluation Units, in relation to actual implementation. Thematic Steering Groups have also been established. In addition, each OP Evaluation Unit can establish its own OP Evaluation Steering Group. The NEU and OP Evaluation Units also co-operate with the NDP and OP Monitoring Committees, whose main role is to discuss and accept recommendations from evaluation reports, to monitor their implementation, to approve evaluation plans and suggest additional evaluations according to strategic or management needs. Detailed Monitoring Committee responsibilities can vary between different OPs. 11

12 Hypothesis of the study Based on evaluation guidelines one can conclude that Polish evaluation system is a coordinated and decentralized with focus on a participative approach. It is considered a decentralized system because the responsibilities for evaluation are within many institutions involved in the process of EU Funds management i.e. NEU, evaluation units in MAs and sometimes even at IB level. Direct coordination is assured by evaluation guidelines. They envisage: coordination within the planning process (all yearly evaluation plans are sent to the NEU), submission of all evaluation reports to the NEU. There is also a system for managing evaluation findings and recommendations. Soft coordination is provided by evaluation capacity building activities such as: training events and post-graduate studies, conferences, methodological papers. As stated earlier, the Polish evaluation system is also participative. This opinion is based on the following: the organisational structure is decentralised thus there are Evaluation Units at every management level (Managing Authorities, Intermediate Bodies) which are coordinated by the NEU - therefore all the structures involved in the management of SF `participate` in the evaluation process; the management of Polish evaluation system is based on the work of 6 thematic steering groups and 1 coordinating steering group. Their task is to bring together representatives of all the managing structures to `participate` in the programming, monitoring and implementation of recommendations from evaluations as well as in the facilitation of evaluation activity itself; the implementation of recommendations from evaluations is monitored by the Monitoring Committees as well as by the NEU. Since Monitoring Committees are by definition participative structures of stakeholders, the main actors concerned are therefore involved in the evaluation process; evaluation activities are mostly externalized, hence there is significant participation of the private sector in the actual implementation of evaluations. One beneficial aspect of this externalisation is the ability to try innovative methodological approaches in evaluations; evaluation activity within the Polish administration is facilitated through the participation of respected academic institutions via postgraduate courses for civil servants and scholars involvement in evaluation studies. 12

13 Such coordinated, decentralised and participative characteristics might even be representative of an ideal evaluation system. The strongest point of a decentralised evaluation system is the fact that evaluation is as close as possible to information and management needs in other words - decentralised bodies can become involved more in evaluation and they feel the ownership of findings and recommendations. As far as the implementation improvement function of evaluation is concerned, decentralisation is an advantage. However it could be less advantageous when strategic planning improvement and accountability functions are taken into account, because implementation of strategic recommendations may not fall within the scope of decentralised management bodies competencies. In order to limit the disadvantages of a decentralised system, a coordination function is put in place. Coordination in this context is to ensure amongst other things: gathering and making available all the information in one place, a common approach to the evaluation process. The participative approach is a strength from an operational perspective because the recommendations are issued close to the point of implementation and are developed with the participation of the people responsible for the follow-up process. In addition, from a strategic perspective the participative approach is a strength because it involves the decision makers in all managing structures. However since strategic decisions are sometimes taken at levels higher than managing structures of EU funds there is a need to also involve those levels. The participative approach is also a strength from the accountability point of view, given that people responsible for managing EU funds take part in both programming and follow-up of the results of evaluations. The table below sums up the above-mentioned characteristics of the evaluation system (decentralised, coordinated and participative) in the form of a SWOT analysis. Characteristic/ Main evaluation functions Co-ordinated decentralised system Participative approach Implementation improvement Strength Strength or Opportunity 2 Strategic planning improvement Strength (could be Threat) 1 Strength or Opportunity 2 Accountability Strength (could be Threat) 1 Strength or Opportunity 2 The analysis suggests that a decentralised, coordinated and participative evaluation system, as applied in Poland, could be taken to constitute best practice in this field. 1 Depends on the field. Could be a threat if the system of managing strategic implementation is not fitted to the strategic recommendations or other systematic measures are taken in order to follow-up strategic recommendations (e.g. systematic meta-evaluations carried out within all fields). 2 Depends whether it is only on paper, or exists in reality. 13

14 Verification of the hypothesis In order to verify the described hypothesis (that the Polish evaluation system, given its decentralized, coordinated and participative characteristics, could constitute a model of best practice), an assessment of all phases of evaluation was conducted. The diagram below shows the assumptions of evaluation lifecycle. Verification of the hypothesis involves answering the central question of whether there is sufficient coordination and participation in each of these stages. Evaluation planning Planning is the first step in the evaluation process. It is essential for achieving the evaluation goals at strategic and operational levels. After performing the desk research on the planning process, it could be established that: the obligation to prepare Evaluation Plans was introduced in the Polish evaluation system from the beginning of the programming exercise, there are two types of Evaluation plans: strategic and operational. The operational evaluation plans indicate the specific activities to be performed in a given year, including capacity building activities. The strategic evaluation plans are prepared for the entire programming period and contain the general areas of evaluation and the organisation of the implementation systems. Having two layers of planning can be an effective approach, if properly put into practice. The interviews indicated that the Multi-Annual Evaluation Plans (MAEP) are drafted at the beginning of the programming period but do not appear to be used afterwards. The MAEP is essentially a general framework and does not include individual evaluation projects to be implemented by the MA. 14

15 Individual evaluations are performed based on Annual Evaluation Plans (AEP). However, these do not provide a high level of detail. For example, only the scope of the evaluations is mentioned in the EAP - not the evaluation questions. Such flexible planning could be a strength only in relation to the operational evaluation, but it represent a threat for a longer perspective (strategic) planning. This was confirmed by interviews and focus group discussion. In terms of coordination, the general perception is that the planning process could still be improved at all levels. It is performing well in theory, but not in practice. It is hampered by factors such as the system structure, the orientation to immediate needs - mainly due to the lack of time and human resources (around 70 people are in charge of evaluation). For example, inside the MA for OP Innovative Economy, the Evaluation Unit consists of only one employee with responsibilities dedicated solely to evaluation. This person is expected to coordinate 5 Evaluation Units from the IBs, but due to the lack of personnel only the most urgent needs can be addressed. As a result, only one Evaluation Unit from the IBs, which is new, is currently coordinated whilst the others are performing evaluation activities without coordination. In the MA for OP HC the situation is better in terms of coordination and participation, but the collaboration between IBs seems to be a problem. In addition both the MA for OP HC and the MA for OP IE have pushed for more horizontal coordination with the regional MAs. However, the collaboration with the NEU is seen as a strong point by the MA for OP Infrastructure and Environment, which receives substantial support for developing all evaluation actions. Staff from the Evaluation Unit in the MA OP I&E stated that they have good collaboration with all the structures involved in their work (e.g. with the public procurement unit). The NEU acknowledged deficiencies in planning coordination were a trade off in satisfying more immediate needs (e.g. preparation of evaluation guidelines for , performing ex-ante and ex-post evaluations, conducting horizontal evaluations). Lack of time and human resources hinder participation in the planning phase of the full evaluation system. In terms of coordination of evaluation plans, the Steering Groups (SGs) fulfilled their role only at the beginning of the process. Now they are either not consulted on evaluation plans or their input is considered more of a formality. The SGs at OP level have more meetings than the SG at the level of NSRF. These are generally informal meetings in which only a few members make significant inputs on ToR, methodologies or reports. An exception is the SG for OP HC, which is active in terms of regular meetings and capacity building. Whilst decentralization at the operational level is considered a strength, it depends heavily on human resource capacity. This could be seen as a weak point particularly where lower level structures are concerned. Some 2 nd level IBs (e.g. under OP Infrastructure and Environment, or OP Innovative Economy) with only a few people in their Evaluation Units, are either performing evaluation without coordination or not performing evaluation at all. Moreover in the case of strategic evaluation, decentralization is more of a weakness since strategic issues are usually better addressed in a centralized manner. 15

16 To sum up regarding the level of coordination and participation in the planning phase of evaluation, the main gap identified is one of insufficient collaboration: - between the NEU and MAs in terms of coordination of planning - in order to avoid overlapping (this is also valid in terms of consultations on ToR, where comments are very few), - between different MAs (sectoral and regional) - this can be improved to avoid overlapping and ensure complementarity and synergy, - between IBs, - even within one MA (e.g. HC OP MA), - within Steering Groups. In terms of effectiveness and usefulness, flexible planning with no strong coordination can sometimes be a strength. However in terms of efficiency it is more often a weakness overlapping causes double work and does not ensure complementarity and synergy of the evaluation (building knowledge). In all cases, lack of human resources and associated time constraints invariably pose a serious threat. Preparation of ToR and commissioning of evaluations Inside the NEU, ToR are developed in accordance with the following steps: the person responsible for the evaluation drafts the ToR; the person responsible consults with the other persons within the NEU (1 st consultation); the NEU consults the persons within the thematic steering group, as well as with additional people from MAs and IBs concerned by the evaluation (2 nd consultation); If the evaluation study requires data which are only available to particular parties, these entities are also consulted (3 rd consultation). In practice the 2 nd consultation is carried out directly with the persons concerned via , if time allows. Sometimes there is no consultation particularly in cases where there is a lack of human resources and serious time constraints. The NEU mentioned that entities downstream follow more or less the same process, only that they consult with IBs and structures at their level concerned by the evaluation study. This was confirmed in the interviews with central and regional MAs. A long-awaited Guidebook for ToR development has recently been issued by the NEU. This includes guidance on: - criteria - evaluation questions 16

17 - methodology requirements - budget calculation - schedule - dissemination etc. The guidebook will be updated whenever the need arises. The NEU mentioned that there are no external factors hindering ToR development. At NEU level, ToR contents were said to be rarely contested and only few significant requests for clarification received. The regional MAs mentioned that they also had to provide very little clarification. There were more requests for clarification noted at the beginning of the programming period, but these are now rare. This was said to illustrate increased experience overall. Regional MA representatives mentioned that they participated in post-graduate studies which included the topic of ToR preparation and public procurement. Even though there were few clarification requests on ToR, the NEU commissioned a study to analyse the tendering process (including ToR) and draw conclusions and recommendations. The tendering process itself is performed by procurement units outside the control of evaluation units. There was felt to be poor collaboration between them due to different interpretation of qualitative criteria by the different units. This is a systemic issue which was found to lead to delays in the tendering schedule and hinder the choice of best quality offer. Public procurement units cannot be influenced directly by the NEU or the Evaluation Units at MA/IB level, therefore the only ones that can influence are the higher stakeholders. The NEU mentioned that the market supplies enough offers for tendering purposes but that there are some areas where supply is lower - either because of lack of demand or because they are difficult fields to assess (e.g. transport, environment, integrated measures territorial dimension). Evaluation studies are priced on average in the range EUR. Larger studies like expost evaluation of the period cost more around EUR. In summary the capacity of the NEU, MAs and IBs to draft ToR for evaluations has improved significantly. This seems mainly due to increased experience. Coordination was/is ensured by: trainings on ToR drafting at the beginning of this programming period and post graduate-studies organised together with Warsaw University. Within the commissioning process there exists a systemic obstacle - poor collaboration with public procurement units because of different interpretation of qualitative criteria. As public procurement units cannot be influenced directly by the NEU, no coordination efforts would facilitate this process. However, the NEU has commissioned a study on the tendering process and plans to discuss its findings with the national entity responsible for public procurement, in order to be able to issue a new guide on commissioning. Evaluation management Evaluation studies are managed as follows: kick-off meeting (where participants are open and frank with focus on evaluation perspective, needed data and documents), phone or contacts between evaluator and commissioner, 17

18 progress reports, draft report (quality-assessed in two rounds of consultation 1 st inside the unit and 2 nd with the stakeholders who are usually also members of the evaluation steering group), final draft and approval. Where many institutions are involved in evaluations, the normal procedure is to send the evaluation reports to all of them for consultations on the particular parts concerning each one. These institutions are then requested to provide opinions. Meetings are also held with the evaluators in case of diverging issues. The report quality assessment sheet found during desk research, was discovered from the interviews not to be a commonly-used tool to assess reports. It was, however, mentioned that careful monitoring takes place concerning each evaluation report s achievement of the objectives stated in ToR. The evaluation studies are mostly completed on time and in budget. The NEU considers its own capacity to manage evaluation studies as very good. But it believes at the same time, that all Evaluation Units are insufficiently staffed. When studies include innovative methodologies, the NEU collaborates with academia. This type of collaboration is remunerated (around EUR). The assessment is performed by a Professor or PhD student. The NEU believes that some MAs and IBs are hesitant to use this type of externalisation. The new evaluation guidelines, however, promote such an approach. Within the HC OP there are separate guidelines on evaluation as there are many institutions involved in the process (also 16 regional). The document is called Evaluation step by step. According to this document: Once the contractor is selected based on the provisions of the law, a first meeting to launch the implementation of the evaluation research is envisaged; a large number of stakeholders should attend the meeting; detailed specifications are provided regarding the scope and the technical and administrative aspects of this meeting. The commissioner should notify all entities involved in the research of the beginning of each stage of its implementation and provide updates on a regular basis, in order to ensure good communication and provide useful support to the evaluator. During the first meeting a Detailed Description of the Subject Matter of the Order is drawn up and sent to the attendees for comment. It includes: scope of research, objective, users of the evaluation, problems (based on the evaluation questions), criteria to be used in carrying out the evaluation process, research methods and requirements for the presentation of the results. Collection of documents and data to be submitted to the contractor should take place on a regular basis. This is considered to be a good practice to secure a smooth work flow and as a prerequisite for a timely provision of the expected deliverables. The Contractor is expected to submit a management report in case of lengthy and extensive evaluation reports. A short memorandum is drawn up after each meeting with the contractor, were this is requested by the contractor, in order to summarize the most important aspects discussed. 18

19 Close cooperation should take place between the contractor and the commissioner in implementing surveys, which are often used as research tools. The commissioner supports the contractor in cases where the response rate to such surveys is too low. The Guidance document has a strong position on maintaining the neutrality of the evaluation report - particularly the conclusions and recommendations - specifying that any discussion on evaluation reports should be limited only to facts. Due to the high complexity of the evaluation process and in order to secure a good quality of the deliverables, MAs should consider designating a reviewer an expert recognized in the evaluation environment, dealing with the issues covered in the research - who will provide an opinion on the report prior to its submission to the commissioner. The final step in the evaluation process is the acceptance of the final report. In the first instance, acceptance of the final report consists of verification that the content of the report is compatible with the description of the subject matter of the Order and methodological approach it describes. This must be validated by an Evaluation Unit of the institution commissioning the study. The final evaluation report is then submitted to the higher institutional level. It is essential that the report presents a complete list of the conclusions and recommendation. Based on the document called Evaluation Capacity in Poland 3, the following relevant aspect was outlined on working with evaluators: The evaluators capacity is assessed by the NEU which sends to Evaluation Units questionnaires asking about reports and the quality of co-operation during external evaluations. They grade the quality of final reports on a 1 to 5 scale (where 1 means very poor, 2 poor, 3 sufficient, 4 good, 5 very good) and the cooperation with the evaluators on a 1 to 4 scale (where 1 means very bad, 2 bad, 3 good, 4 very good). Over 59% of the commissioning institutions have assessed the reports from evaluations conducted so far as good and very good, fully meeting their expectations. Only 4.4% of the reports are assessed as poor. 31.1% are assessed to be sufficient. A considerable growth in the quality of the evaluation reports can be observed. In 2007, the number of evaluations assessed as good or very good increased considerably (from 24 to 40) in comparison to 2006, whereas the number of reports assessed as sufficient or poor was halved. It can be concluded that after the initial stage, once the evaluators gained experience, the quality of the reports increased. Cooperation with the evaluators was assessed even higher almost 85% of the evaluation promoters assessed it as good and very good. Less than 10% of them were not satisfied with the cooperation. As in the case of the quality of evaluation reports, cooperation with the evaluators is also assessed positively with increasing frequency. [Bienias (ed.), Gapski, Guzek, Mackiewicz, Nowak, Opałka, Strzęboszewski, Sudak (ed.), 2008, p ] It appears that the evaluation management process does not need much coordination. Nevertheless, based on desk research, it is clear that the following aspects are envisaged in the monitoring and coordination of evaluations: - participatory approach by involving key stakeholders in monitoring from the beginning and throughout implementation; 3 19

20 - management reports requested from the contractor in order to secure a smooth and coordinated implementation of the evaluation; - consultations which aim to ensure accuracy and realism in evaluations. Follow-up process Decentralization of the evaluation system results in a large number of different institutions conducting evaluation studies. In such a situation the key challenge becomes knowledge management. The first precondition for using existing knowledge is gathering and making available all the information in one place. That is why the NEU to create a database, Evaluation report data base, with all finalized evaluation studies conducted by different institutions and on different (higher than project) levels of implementation. The database is available via Internet and allows for downloading of full versions of final reports. In addition, the database (which is a simple Excel file) allows the user to find the particular report that is needed as it provides a possibility to filter by different categories (area of intervention, OP, date of accomplishment, financial perspective, size of the research, commissioner and evaluator). The database is one of the first and most useful sources of information for evaluators, administration, academic researchers and allows for open access for the public (mostly journalists). 20