Learning from evaluations Processes and instruments used by GIZ as a learning organisation and their contribution to interorganisational learning

Transcription

1 Monitoring and Evauation Unit Learning from evauations Processes and instruments used by GIZ as a earning organisation and their contribution to interorganisationa earning

2 Contents 1.3Learning from evauations as part of GIZ s corporate strategy 3 2.3Learning from evauations as part of GIZ s management mode 4 3.3Learning from evauations so that they are effective at a eves 5 4.3Learning orientation in the monitoring and evauation system 7 5.3Learning from evauations the process Instruments for earning from evauations Sharing knowedge as a contribution to interorganisationa earning Chaenges 14. Imprint Pubished by Deutsche Geseschaft für Internationae Zusammenarbeit (GIZ) GmbH Registered offices Bonn and Eschborn, Germany Friedrich-Ebert-Aee Bonn T F Dag-Hammarskjöd-Weg Eschborn, Deutschand T F E evauierung@giz.de I Responsibe Martina Vahhaus Authors Sandra Fuhr and Dorothea Giesen-Thoe Design concept and ayout Iris Christmann and Christian Mentze Photo credits Dirk Ostermeier Charts Additiv Visuee Kommunikation Berin (page 4) Katrin Straßburger (page 5) Lithography Andreas Groß Printed on 100% recyced paper certified to FSC Copy deadine Apri 23, 2013, Bonn and Eschborn 02_Learning from evauations

3 This strategy paper is based on the understanding that the Deutsche Geseschaft für Internationae Zusammenarbeit (GIZ) is a earning organisation, which means that earning and further deveopment are key factors for its success. Against this backdrop, the pubication describes the process of earning from evauations within the framework of GIZ s evauation system and expains the instruments used in this context. It expains how the findings of evauations are incorporated into GIZ s corporate processes and used for organisationa earning. It aso demonstrates how GIZ shares its knowedge about conducting evauations and the findings estabished in this context, in order to contribute to interorganisationa earning. This pubication is primariy directed towards speciaists who dea with the essons earned from evauations within the context of quaity management systems in organisations and companies. 1. Learning from evauations as part of GIZ s corporate strategy GIZ s vision is to become the word s eading provider of internationa cooperation services for sustainabe deveopment. It faces the chaenge of incorporating earning and knowedge into its strategies as key factors for promoting innovation and competitiveness, in order to support peope and societies around the gobe in shaping their future and sustainaby improving their iving conditions. It does this using an efficient, effective and partner-oriented approach. In order to achieve this goa, GIZ wants to utiise and buid the earning potentia and knowedge of its empoyees and of the organisation as a whoe, in order to increase the company s performance capacity and effectiveness. As a earning organisation, GIZ aims to use evauation findings for the company-wide earning process. Evauations are a key instrument for identifying evidence-based resuts of internationa cooperation, in order to meet accountabiity obigations towards commissioning parties and cients, partners and the genera pubic. But that is not a. They aso aow us to systematicay refect on essons earned, in order to estabish what works in the ong term and what doesn t. With 100 evauations each year, GIZ and its partners can everage significant earning potentia to continuay improve its work. GIZ has deveoped specia processes and instruments for channeing findings into the continuous improvement process. Learning from evauations _03

4 2. Learning from evauations as part of GIZ s management mode 1 For more information on Capacity WORKS, go to: ourservices/1544.htm Capacity WORKS 1, GIZ s management mode for sustainabe deveopment, is used to steer projects and programmes. Success factor Learning and innovation covers questions such as: What have we earned from evauations? How do we communicate the essons earned to our partners and channe them back into the company? How do we further enhance our effectiveness? Figure 1: The five Capacity WORKS success factors To answer these questions, GIZ aso focuses on promoting earning skis through an effective knowedge management system. The aim of this system is to optimise the efficiency with which knowedge and essons earned are exchanged in the management processes competence buiding and experience-based earning. This invoves refecting on, anaysing and evauating findings and systematicay buiding our knowedge to equip ourseves better for future tasks. 04_Learning from evauations

5 Quaity management at GIZ is based on the premise that the objectives system is geared to the guiding principe of sustainabe deveopment. This means that evauation is one of GIZ s quaity features and is used to identify and everage the potentia for optimising resuts, in order to safeguard and enhance quaity in the vaue GIZ s work generates 2. 2 For more information on GIZ s Quaity House, go to: quaity_management.htm Figure 2: GIZ s Quaity House 3. Learning from evauations so that they are effective at a eves...and hep achieve sustainabe deveopment. It is not easy to optimise the potentia of evauations, as organisations tend to have fixed structures that are difficut to change. If individuas often find it hard to earn from experience, particuary from mistakes, it is sma wonder that the obstaces to earning are even more difficut to overcome at the project or corporate eve. This is exacerbated by the fact that evauations are frequenty perceived more as a monitoring instrument and as additiona effort rather than as an opportunity for earning. However, it is essentia that a company such as GIZ which has a decentraised organisationa structure, a fied structure and high staff turnover earns from evauations at a eves. This is the ony way that GIZ can increase the quaity and effectiveness of the services it provides and boost its competitiveness in the ong term. In order to meet this chaenge, it is vita that we understand the theory of organisationa earning and appy it in the corporate context. Learning from evauations _05

6 At GIZ, earning takes pace in oops between three different eves: 1. At the individua project or programme eve, staff members, externa experts and nationa partners are invoved in evauations. At a ater stage, these stakehoders act as mutipiers for important findings. 2. At the eve of the organisationa units (usuay the Sectora Department and regiona departments), evauation findings are examined beyond the individua project or programme to improve sectora and regiona approaches. 3. At the institutiona or corporate eve, seected recommendations for action must be consistenty integrated into ongoing business processes, in order to faciitate organisationa earning to modify strategies, processes and instruments. Figure 3: Learning from evauations Adapted from Argyris and Schön s oop mode (Ramaingam, B.: Organisationa earning for aid, and earning aid organisations, in: Capacity.org, Issue 33 (2008), S.4) Company Organisationa units Organisationa units Evauation meeting Project/ programme Project/programme evauation Evauation meeting Workshop Resuts EVALUATION Corporate strategy evauation Cross-section evauation Learning café Executive management decision Management Response System > Impementation agreement based on evauation recommendations Evauations deiver evidence-based findings that enabe action-oriented earning from experience within the company at the different eves of the tripe-oop earning mode. Initiay, knowedge is acquired at the project eve. In order for a company to benefit from this knowedge, it has to be integrated into further processes throughout the company. This requires exchange at different eves: horizontay between the different organisationa units within a company, and verticay between the different hierarchica eves, from the management to the individua empoyees. The key task in this context invoves processing experience so that different actors can earn from it through different processes. 06_ Learning from evauations

7 In order to channe evauation findings beyond the individua project/programme and make them avaiabe for GIZ and its business processes, the Monitoring and Evauation Unit has deveoped various evauation and earning toos, which are described in greater detai in sections 5 and 7. These toos institutionaise earning from evauations by feeding the findings from the three earning eves back into the process. 4. Learning orientation in the monitoring and evauation system GIZ has a number of monitoring and evauation toos to hep identify earning fieds. It distinguishes between decentraised evauations (which incude project progress reviews and are conducted and steered by projects/ programmes on site) and centra independent evauations (which are conducted and steered by the Monitoring and Evauation Unit). The different evauation toos enabe experience-based earning on at east one of three eves (project, organisationa unit or corporate eve). Independent corporate strategy and cross-section evauations that take the form of meta-evauations and evauation syntheses pay a specia roe in this context, because they focus on incorporating knowedge from outside GIZ and new knowedge, based on knowedge that is aready avaiabe in the company, rather than on earning from experience. Decentraised Evauation Steering by operationa units GIZ s Monitoring and Evauation Instruments Centra (Independent) Evauation Steering by the Monitoring and Evauation Unit Figure 4: Monitoring and evauation toos Project Progress Review (PPR) Portfoio evauation Fina evauation Ex-post evauation Other decentraised evauations Cross-section evauation Corporate strategy evauation Comparative perspective assessment (KOMPASS) Resuts-based monitoring (RBM) for continuous provision of reevant data and information Learning from evauations _07

8 3 Resuts-based monitoring Resuts-based monitoring is an interna too used to review project progress on an ongoing basis. It is therefore used for project/programme steering and for reporting. It provides the basis for evauations and contributes to earning in a project or programme. Monitoring findings make it possibe to communicate what works and where improvements need to be made, based on sound evidence. In so doing, they promote the earning process in a project or programme and are incorporated into knowedge management. The essons earned are used to modify the project/programme design or to prepare a new project or programme. KOMPASS Comparative assessment of perspectives Within the context of GIZ s (advisory) work, the opinions and perceptions of the different project stakehoders invoved have a direct bearing on the quaity of service provision and the effectiveness of a project or programme. The views and experience of the partners and target groups are therefore openy and systematicay documented at east once. Data are coected using a quaitative instrument such as the computerbased interview procedure e-val, in order to identify the strengths and weaknesses of a measure from the perspective of different actors. The findings are systematicay assessed, discussed in the project team together with the partners and then incorporated into project/programme steering and design using a Management Response System. Project progress reviews Project progress reviews (PPR) are used to steer and pan deveopment projects and programmes. They combine a critica and anaytica review of resuts with a strategic forecast of future deveopments in a project or programme. PPR resuts are used for steering in conjunction with partners and contribute to panning the continuation of a project or programme. Other decentraised evauations Decentraised evauations are carried out at different stages (for exampe as an interim or fina evauation) and for a variety of reasons. The object of an evauation (e.g. a project/programme, sector portfoio, country portfoio, instrument or product) depends on the information required, the areas of interest and on the expectations identified by the project team in conjunction with the partners before the evauation. The cearer a picture stakehoders have of what information is required, the more usefu the findings. The findings and recommendations of decentraised evauations are shared with a stakehoders and beneficiaries to initiate a earning process. 08_ Learning from evauations

9 Portfoio evauations Portfoio evauations (independent evauations) are fina and ex-post evauations that assess the effectiveness of deveopment project/programmes retrospectivey or at their concusion. A different priority area is evauated each year. By conducting evauations in ine with rigorous methodoogy, GIZ aso responds to the chaenge of substantiating and attributing resuts within the increasingy compex deveopment cooperation setting. The findings are discussed in an evauation meeting. Cross-section evauations GIZ acquires information on cross-programme organisationa earning and on the quaity of evauations from cross-section evauations, which take the form of evauation syntheses/synthesis reports and meta-evauations or reviews. The findings of the different evauations are anaysed and summarised in the synthesis reports. Conducting another comparative assessment based on questions and criteria that have been specificay chosen by GIZ generates new knowedge that enabes cross-programme success factors, potentia for improvements and good practice to be identified. The findings of cross-section evauations are presented to the management body, aowing it to monitor the impementation of any recommendations made and draw concusions and recommendations for the company as a whoe. The Monitoring and Evauation Unit assesses portfoio evauations centray in a cross-section anaysis together with decentraised evauations of the portfoio. Cross-section evauations of decentraised evauations are aso conducted on behaf of and by the reevant division of the Sectora Department. For exampe, a project progress reviews conducted in recent years in a specific sector are evauated, in order to aggregate sector-specific findings. Corporate strategy evauations Corporate strategy evauations address the impementation of poicies and strategies throughout GIZ that reate to the deivery of services. They usuay span different business sectors and instruments. Corporate strategy evauations assess the impementation and appication of GIZ s poicies and strategies and aso examine issues that wi hep deveop them further. This too aims to enabe evidencebased earning and decision-making throughout the company, in order to boost the effectiveness, efficiency and quaity of service provision at GIZ and (further) deveop key poicies and strategies. Learning from evauations _09

10 5. Learning from evauations the process Learning from evauations is a process that is broken down into severa steps. The evauation process receives the strongest support and achieves the best resuts when earning both inside and outside GIZ inks into existing processes and is incuded as an integra eement of project management. After a, such experience-based knowedge can ony be used effectivey if it is fuy integrated into the strategies and processes of projects/programmes and of the company as a whoe. GIZ promotes the foowing earning process from evauations: We ensure that the terms of reference for evauations are geared towards earning needs. When compiing the evauation report, we make recommendations that are action-based and user-centred. We assess evauations across different sectors, regions, instruments and quaity criteria, for exampe, to ensure that findings refect the situation across programmes. We use systematic concusions drawn from the discussion of evauation findings in evauation meetings with participating organisationa units and with evauators, earning cafés or earning workshops with participating organisationa units, the evauators and other organisationa units that can use the findings and recommendations, sector networks in presentations in penary and working groups. We agree within the scope of the Management Response System that evauation findings and recommendations be taken into account in foow-on phases/new projects and programmes, sector strategies, approaches and concepts, product deveopment. The key findings and impementation agreements from cross-section evauations are presented to GIZ s management bodies. We present a summary of these findings to corporate management and the Management Board, aong with any reated recommendations. We document the findings, recommendations and agreements in the Knowedge Base and other search engines that can be accessed throughout GIZ. We communicate the findings, recommendations and agreements using an existing format that is appropriate for the specific recipient.. 10_ Learning from evauations

11 6. Instruments for earning from evauations GIZ has deveoped the foowing earning instruments to promote earning from evauations: 3 Learning and user-oriented terms of reference The process of earning from evauations starts with the terms of reference. The recipients of the evauation findings and recommendations hep identify the issues to be examined in the evauation. In particuar, coeagues from the reevant divisions within GIZ s Sectora Department (interested in technica and sectora issues) and from the responsibe regiona departments (interested in cutura and country-specific issues) are asked to outine what information they need so that this can be incuded in the terms of reference, aong with the interests of managers (concerned with cross-sector issues reated to corporate poicy and deveopment poicy) and of experts from the fieds of evauation and knowedge management, who are mainy concerned with methodoogy. 3 Learning and benefits-oriented reporting grid Evauation reports deiver evidence-based, action-oriented recommendations that hep optimise earning outcomes. Ideay, these recommendations shoud be broken down into different categories, based on the recipients who are intended to earn from them. An annotated or agreed structure is used to ensure that reports are of uniform quaity and that they meet the earning expectations of users. Evauation meetings The findings are discussed in evauation meetings, in order to promote earning outcomes among individua experts and members of project staff. Learning workshops Learning workshops are suitabe for a forms of evauation. They can be conducted foowing an individua evauation, or an evauation of a specific sector or theme, for exampe. The evauation findings and recommendations are discussed with a imited number of representatives of the evauation s commissioning parties, participants, project staff members and reevant managers. The number of peope invoved in this discussion shoud be kept to a minimum, and everyone invoved shoud be famiiar with the object of the evauation. Learning workshops focus on sharing and examining the evauation findings. They bring together the peope who are directy affected by the evauation, enabing them to decide on specific impementation agreements. The aim is to provide a safe space to create an open earning cuture and to earn from mistakes. These workshops are therefore particuary suitabe for earning from projects/programmes that have received a poor evauation rating. Learning from evauations _11

12 Learning cafés A earning café is a form of diaogue and workshop that is particuary suited to arger groups. It aims to enabe the interactive exchange of knowedge, in order to deveop new strategic orientations. Here, evauation findings are discussed in a diaogue forum that spans severa organisationa units. The evauators and experts directy invoved in and responsibe for an evauation as we as a beneficiaries of the evauation findings attend earning cafés. These beneficiaries incude representatives of the regiona divisions and divisions in the Sectora Departments as we as managers from the reevant divisions. Based on the evauation findings, the representatives of different organisationa units poo their expertise on a topic and examine it from different perspectives. This exchange between a recipients who wi earn from the evauation findings ensures that the essons earned wi be used beyond the evauated project or programme, aowing structura deficits in corporate processes to be identified and approaches and procedures to be improved. The effectiveness of strategies is examined and assessed and exempary approaches (best practices) are identified. Management Response System The Management Response System heps projects and programmes to improve quaity and increase effectiveness in a more targeted manner by fostering earning from evauation findings. It provides a structure for working on evauation findings, incuding two tabes that aow evauation recommendations to be documented ceary and succincty. The first step is to take a ook at the recommendations made by the evauators: Are these usefu? Are they based on the anayses of the evauation? Can they be put into practice? A decision is then made on whether they wi be accepted and in what form. If they are accepted, a pan of action is drafted that defines how and in what time frame they wi be impemented. Responsibiities are defined. Progress on impementing recommendations is monitored after one year. Sector networks Sector networks provide a patform for the organised exchange of expertise and for cooperation between GIZ staff members in projects and programmes in partner countries and at Head Office. The networks have a regiona and sectora focus and aim to promote joint experienced-based earning beyond the borders of individua countries. Sector networks identify, discuss and assess the findings of evauations, above a in reation to cross-cutting issues. The knowedge gained in this context is then processed into best-practice exampes, deveoped into products or stored in GIZ s Knowedge Base. 12_ Learning from evauations

13 The foowing tabe outines the earning instruments used in the different evauation toos, in order to support the process of earning from evauation findings. Evauation too Other decentraised evauation Project progress review KOMPASS Independent evauation Cross-section evauation Corporate strategy evauation Instruments for earning from evauations Terms of reference Reporting grids Evauation meetings Learning workshops Learning cafés Management Response System Sector networks 7. Sharing knowedge as a contribution to interorganisationa earning Through its transparency and information poicy 4, GIZ aims to strengthen the exchange of knowedge and information. The objective is not just to provide the genera pubic with data and documents on GIZ s work and resuts, but aso to engage in an intensive exchange of information with oca partners, with commissioning parties and cients and with cooperation partners. GIZ aims to shape interaction between peope with diverse events and diaogue forums, to enabe sharing of knowedge and experience with others. 4 For more information on GIZ s transparency and information poicy, go to: transparency.htm By communicating and pubishing information from and about evauations, GIZ aso promotes the exchange of knowedge and experience with other internationa cooperation institutions, universities and research institutions as we as with private sector and civi society organisations (companies, business associations, NGOs, poitica foundations etc.). This benefits not ony GIZ s cooperation partners, but aso GIZ itsef. On the one hand this is how we disseminate our approaches, topics, essons earned and attitudes, and in so doing further enhance our reputation in the internationa debate. On the other hand we aso earn from the experiences of others. This heps us avoid mistakes, innovate and further strengthen the effectiveness of our projects and programmes. Learning from evauations _13

14 Cooperation activities are especiay important as a means of transferring knowedge and expertise, accessing new knowedge and cosing knowedge gaps. When cooperating, GIZ uses persona contact to share its knowedge with the actors concerned directy and in great detai. To date, GIZ has cooperated primariy with the Center for Evauation at Saarand University and with Wageningen University. GIZ reguary pays an active part in internationa forums (e.g. evauation conferences), and pubishes most of the resuts of its work in appropriate pubications or onine. It is activey invoved in networks and invites third parties (evauators, academics, organisations and companies) to participate in events on evauation findings and methodoogies. 8. Chaenges In order to increase the effectiveness of its work, GIZ must earn from evauation findings and channe this expertise into its day-to-day activities. Existing evauation toos pay a key roe in supporting earning at a eves throughout the entire company. However, earning oops are ony reay competed when the assessment of evauation findings has succeeded in bringing about change. There are sti a number of chaenges to be met, in order to fuy institutionaise earning from evauations at GIZ. Learning takes pace when the focus of an evauation shifts from the expected to the unexpected and expores new territory rather than justifying why something did or did not happen, examining the findings, concusions and recommendations, and the changes that need to be impemented on this basis. The greater the degree to which findings and recommendations are geared to the information needs identified by a project or programme, the more open the process to assess evauation findings wi be. Evauations shoud concentrate on the issues most reevant to a project or programme. Furthermore, the earning potentia from evauations wi be optimised when a stakehoders are prepared to transate new findings into action, and to earn from mistakes. In particuar, an open corporate cuture is vita if we are to earn from our mistakes. Consequenty, the creation of an atmosphere of trust is mainstreamed in GIZ s Guiding Principes for Managers. The Management Response System documents impementation agreements that are binding, and tracks the impementation of evauation recommendations. One chaenge that is faced in this context, however, is the targeted use of evauation findings when panning new projects and programmes in GIZ s fied structure. 14_ Learning from evauations

15 The partner institutions of projects and programmes are invoved in a evauations from the outset. This process starts when the evauators terms of reference are drafted and usuay ends with their debriefing and the submission of the report. To give rise to ownership, a methodoogica understanding of the procedures invoved in evauation is an advantage. This wi aso hep ensure that evauation findings are deat with constructivey and support earning. For this reason, capacity deveopment in the fied of evauation is one of the chaenges that needs to be met within the scope of invoving partners and training GIZ staff members. A staff members who dea with commission management shoud therefore receive appropriate training as part of GIZ s interna training programme. The Evauation Capacity Deveopment Programme impemented by GIZ aso aims to strengthen evauation capacities in partner countries. Learning from evauations is worthwhie! We are not there yet, but we are getting there. We are happy to share know-how with you about how to proceed, and ook forward to our further exchange of ideas and experience. Learning from evauations _15

16 Deutsche Geseschaft für Internationae Zusammenarbeit (GIZ) GmbH Registered offices Bonn and Eschborn Bonn und Eschborn Friedrich-Ebert-Aee Bonn, Deutschand T F Dag-Hammarskjöd-Weg Eschborn, Deutschand T F E evauierung@giz.de I