Types and Functions of Evaluations

Size: px

Start display at page:

Download "Types and Functions of Evaluations"

Gary Watts
3 years ago
Views:

1 Types and Functions of Evaluations in Three Austrian Sustainability Strategies Ralf Nordbeck, Michael Pregernig, Karl Hogl 1

2 Types of Evaluations Progress Monitoring: Output-type monitoring and reporting Progress reporting Spending reviews Auditing Formative and Summative Evaluations: Process evaluation Implementation evaluation Impact evaluation Outcome evaluation 2

Auditing Formative and Summative Evaluations: Process

3 Different Functions of Evaluations Planning Function Economic Function Political Function Administrative Function Decision-making: develop and evaluate decision alternatives Implementation: inform and motivate persons involved Monitoring: examine outcomes and effectiveness Feedback and adaptation: provide basis for learning from evaluation results Optimization: safeguard efficient allocation of resources rationalist-instrumental functions Legitimation: provide a source of authority for political decisions and actions Accountability: the allocation or acceptance of responsibility for actions Whitewash: cover up programme failure Postponement: provide arguments to avoid or postpone action Ritual use: evaluation without any intention to use the findings Survival: provide arguments for continued existence of institution or process Growth: provide arguments for securing resources Territoriality: provide arguments for protecting administrative competencies Self-presentation: document administrative willingness to learn political, symbolic functions

of resources rationalist-instrumental functions Legitimation: provide a source of authority for political decisions and actions Accountability: the allocation or acceptance of responsibility for

4 Discussion Which type(s) of evaluations have been used in the three Austrian strategy processes and what are their functions? How to explain the different types and functions of evaluations in the strategy processes? 4

5 Strategy processes as elements of environmental governance Key tenets of environmental governance: (Jänicke & Jörgens 2006) Strategic approach: consensual, broad-based target and strategy formulation with a long-term horizon Integration: of environmental concerns into other policy areas and sectors Participation: by NGOs and citizens Cooperation: between state and private-sector actors Multi-level coordination: from global to local Monitoring and evaluation: diverse range of reporting obligations and indicators National sustainable development strategies: transition form misconceptions of ideal and static master plans and one-off initiatives, to sets of co-ordinated mechanisms and continuing processes of monitoring, learning and improvement. (Dalal-Clayton & Bass 2002) 5

actors Multi-level coordination: from global to local Monitoring and evaluation: diverse range of reporting obligations and indicators National sustainable development strategies: transition form

6 3 Austrian strategy processes Austrian Biodiversity Strategy Austrian Forest Dialogue Austrian Strategy of Sustainable Development 6

7 Austrian Biodiversity Strategy Main Types of Evaluation Progress Monitoring National Biodiversity Commission: central role as information pool & in monitoring of implementation activities Online database Living Document : collection of on-going activities 501 activities reported as of Feb. 2003, but no further activities added since then Evaluation Two-step procedure: 1 st evaluation (2001) assessed to what degree the measures formulated in the 1 st ABS had already been implemented and by whom 2 nd evaluation (2003) assessed the strategy document itself 7

8 Assessment of the ABS Evaluation 1/2 Type: mix of process and output evaluation semi-outsider evaluation (Federal Environmental Agency) partly based on self-reporting Functions: 1 st evaluation: Ministry of Environment wanted to keep the topic of biodiversity on the national political agenda (administrative function) work-to-rule exercise to fulfill international obligations (political function) 2nd evaluation: was supposed to lay foundation for revision of ABS (planning function) 8

the topic of biodiversity on the national political agenda (administrative function) work-to-rule exercise to fulfill

9 Assessment of the ABS Evaluation 2/2 Impact: 2 nd ABS (2005) partly based on evaluation results some of the most severe flaws of the first strategy could be cured new type of implementation instrument was introduced (Action Plans) but 2 nd strategy still falls far behind an ideal-type strategic planning approach Institutional and Governance Context: 1 st ABS adopted by Council of Ministers, 2 nd only by Commission Biological diversity as a poorly comprehensible term Low problem pressure and lack of public attention Low political weight and weak institutionalisation Complex allocation of competences and high number of veto players Antagonistic stakeholder interests: a weak environmental coalition and strong business interests groups 9

adopted by Council of Ministers, 2 nd only by Commission Biological diversity as a poorly comprehensible term Low problem pressure and lack of public attention Low political weight and

10 Austrian Forest Dialogue Main Types of Evaluation Progress Monitoring Internal process surveys in the course of programme formulation Internal monitoring of implementation by BMLFUW Evaluation No external evaluation so far (to be decided by mid 2008) 10

formulation Internal monitoring of implementation by BMLFUW

11 Assessment of the AFD Evaluation 1/2 Type: Internal process surveys among participants: evaluating and monitoring the process Internal (informal) monitoring of Work Programme implementation by BMLFUW Functions: Internal process surveys - keeping actors on board / the process on track (political function): participants attitudes and perceptions as regards procedural aspects; internally used by process management bodies Internal monitoring of Implementation for keeping track (planning function) 11

keeping actors on board / the process on track (political function): participants attitudes and perceptions as regards

12 Assessment of the AFD Evaluation 2/2 Impact: Procedural management partly influenced by results of process surveys (topics to the agenda, priority of measures, AFD goals) No outcome evaluation so far Institutional and Governance Context: Forest Programme and Work Programme adopted by Round Table in 2005; Council of Ministers took notice Moderate institutionalisation (in comparison) No specific budgets for implementation of measures Low public attention to forest policy Distribution of competences among federal level and provinces Resistance of traditional forest policy actors (e.g. to discuss Forest Law) 12

2005; Council of Ministers took notice Moderate institutionalisation (in comparison) No specific budgets for implementation of measures Low public

13 Austrian Strategy for Sustainable Development Main Types of Evaluation Progress Monitoring Working Programmes (WP 2003, WP 2004) Online Database of Measures (currently 281 measures) Bi-Annual Progress Reports (2004, 2006) Bi-Annual Indicator Reports (2004, 2006) Semi-external process and implementation reviews Evaluation External evaluation done by independent researchers and consultants in 2005 Review by the Austrian Court of Audit (October-November 2005)

Bi-Annual Indicator Reports (2004, 2006) Semi-external process and implementation reviews Evaluation External

14 Assessment of the ASSD Evaluation 1/2 Type: Mix of progress reviews and process/impact evaluations Evaluations focused on implementation instruments of the ASSD, not on the strategy and its policy goals External evaluations (Independent researchers; Court of Audit) Functions: Evaluation as part of the strategy process ( Learning strategy ) (planning function) Present ASSD as successful role model; Fulfill reporting obligations under the EU SDS (political functions) 14

(Independent researchers; Court of Audit) Functions: Evaluation as part of the strategy process ( Learning strategy )

15 Assessment of the ASSD Evaluation 2/2 Impact: Recommendations will be used to strengthen the strategy process Stronger position for the Forum Sustainable Austria More active role of the federal states ( Regionalization ) Prioritization of the Working Programmes and Measures Institutional and Governance Context: Supportive international and domestic political context ASSD has been adopted by Council of Ministers in 2002 High degree of institutionalization No capacity problems (staff and financial resources) 15

Programmes and Measures Institutional and Governance Context: Supportive international and domestic political context ASSD has

16 Comparative Perspective Type Function Institutional Capacities and Political Profile ABS Process and Output Evaluation Mainly Adminstrative Low AFD Process Reviews Mainly Political Moderat ASSD Process, Output and Impact Evaluation Planning+Political High

Adminstrative Low AFD Process Reviews Mainly Political Moderat

LUKHANJI MUNICIPALITY PERFORMANCE MANAGEMENT FRAMEWORK

LUKHANJI MUNICIPALITY PERFORMANCE MANAGEMENT FRAMEWORK INTRODUCTION The Municipal Systems Act, 2000, which requires a municipality to establish a performance management system that is: Commensurate with