Basic Concepts in Monitoring and Evaluation

Transcription

1

2

3 Basic Concepts in Monitoring and Evaluation February 2008 Published in the Republic of South Africa by: THE PUBLIC SERVICE COMMISSION (PSC) Commission House Cnr. Hamilton & Ziervogel Streets Arcadia, 0083 Private Bag x121 Pretoria, 0001 Tel: (012) Fax: (012) Website: National Anti-Corruption Hotline Number: (Toll-Free) Compiled by: Branch Monitoring and Evaluation Distributed by Directorate: Communication and Infomation Services ISBN: RP: 14/2008

4 FOREWORD iv CHAPTER 1 INTRODUCTION The purpose of this document Intended audience Definition of Monitoring and Evaluation Importance of Monitoring and Evaluation Purposes (and uses) of Monitoring and Evaluation Outline of this document 6 CHAPTER 2: THE CONTEXT OF MONITORING AND EVALUATION The Context of the Developmental State Location of M&E in the policy process Location of M&E in the planning process The Government-wide M&E System Important institutions with a role in M&E Departments at the centre of government on the national level 14 The Presidency 14 National Treasury 14 Department of Public Service and Administration (DPSA) 14 Department of Provincial and Local Government (DPLG) 15 Statistics South Africa (Stats SA) 15 South African Management Development Institute (SAMDI) Departments at the centre of government on the provincial level 15 ii Line departments 15

5 2.5.4 Constitutional institutions 15 Public Service Commission (PSC) 16 Auditor-General 16 Human Rights Commission 16 CHAPTER 3: EVALUATION PERSPECTIVES The idea of evaluation perspective The Balanced Scorecard of Kaplan and Norton Programme performance perspective Financial perspective Governance perspective Human Resource Management (HRM) perspective Ethics perspective The perspectives adopted by National Treasury Guidelines 23 CHAPTER 4: VALUES The value basis of monitoring and evaluation Deriving standards of performance from values The values and principles governing public administration The Eight Principles of Batho Pele Other concepts/ principles expressing some dimension of public service performance 35 CHAPTER 5: EVALUATING PROGRAMME PERFORMANCE Programme evaluation Logic models Results-Based Management Theory-based evaluation 44 iii

6 CHAPTER 6: APPLYING THE CONCEPTS Focusing monitoring and evaluation Designing monitoring frameworks Framing evaluation questions Examples of monitoring and evaluation of different dimensions 54 CHAPTER 7: SUMMARY 60 LIST OF SOURCES CONSULTED 64 INDEX 68 Figures 1 The policy life cycle 9 2 Planning and review cycle 10 3 Policy frameworks of the Government-wide Monitoring and Evaluation System 12 4 Components of the logic model 42 5 Example of an outcome chain 45 Tables 1 Deriving indicators and standards from values 27 2 Examples of objectives and indicators according to the logic model 50 3 Examples of evaluation questions: Housing programme 52 4 Types of monitoring and evaluation in relation to evaluation perspectives and values 55 Boxes 1 Example of a theory underlying causal relationship between outputs and outcomes 29 iv

7 2 Examples of evaluative findings with regard to the efficiency of programmes 30 3 Examples of evaluative findings with regard to the effectiveness of programmes 36 4 Examples of evaluative findings with regard to the sustainability of programmes 37 5 Examples of evaluative findings with regard to the scale of engagement of programmes 37 6 Examples of evaluative findings with regard to the targeting of programmes 40 7 Illustrations of types of evaluations: Housing programme 44 8 Examples of a theory underlying the causal relationship between outputs and outcomes 45 9 Illustrations of types of evaluations: Housing Programme 58 v

8 Foreword The growth of Monitoring and Evaluation (M&E) units in government, together with an increased supply of M&E expertise from the private sector, calls for a common language on M&E. M&E is a relatively new practice, which tends to be informed by varied ideologies and concepts. A danger for government departments is that these diverse ideological and conceptual approaches can exacerbate confusion and misalignment. The standardisation of concepts and approaches in government is particularly crucial for the enhancement of service delivery. The PSC s mandate requires of it to monitor and evaluate the organization and administration, and the personnel practices, of the Public Service. Taking this mandate and the need for a common understanding of concepts and approaches into account, the PSC decided to produce this text on basic M&E concepts. A very basic question asked when a monitoring system must be developed or when an evaluation is planned is: What to monitor or evaluate, that is, what should the focus of the monitoring or the evaluation be? This document tries to answer this basic question by introducing concepts and frameworks. Evaluation involves a value judgement. Many of the concepts discussed in the document have the status of values. The PSC has the specific constitutional responsibility to promote the application of these values in the Public Service. This document is by no means definitive, but the PSC hopes that it will contribute to better understanding and enriched debate about the utility of M&E as a tool for improving the performance of the Public Service. We hope that it will fill the gap that currently exists for an accessible document that caters for managers in the Public Service, whilst also providing a common point of reference to the more advanced practitioner. It is hoped that readers will feel compelled to delve more deeply into the discipline. I trust that this document helps you to deepen your interest and understanding of monitoring and evaluation. Yours sincerely vi PROFESSOR STAN S SANGWENI CHAIRPERSON: PUBLIC SERVICE COMMISSION

9 Chapter One Introduction 1

10 INTRODUCTION This Chapter deals with the following: Purpose of this document Intended audience Definition of monitoring and of evaluation Importance of monitoring and evaluation Purposes (and uses) of monitoring and evaluation Content outline 1.1 The purpose of this document The purpose of this document is to clarify basic M&E concepts and ideas as they apply in the context of the SA Public Service; put the concepts in a framework showing the interrelationships between them; contribute to the development of a coherent and dynamic culture of monitoring and evaluation in the Public Service; and contribute to a better understanding and enriched debate about the different dimensions of public sector performance. Hopefully this document will complement and enhance the work being done as part of the Government-Wide Monitoring and Evaluation System, particularly the Framework for Managing Programme Performance Information (published by the National Treasury in 2007) and the Policy Framework for the Government-Wide Monitoring and Evaluation System (published by the Policy Coordination and Advisory Services in the Presidency, in 2007). 1.2 Intended audience This document is intended for use by M&E practitioners; Senior Management in the Public Service; and managers of service delivery units, who produce performance information and statistics, and who are also required from time to time to reflect on and evaluate the success of their work. 2

11 1.3 Definition of Monitoring and Evaluation Monitoring and evaluation have been defined as: Monitoring Evaluation A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds 1. The systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or programme. An assessment, as systematic and objective as possible, of a planned, on-going, or completed development intervention. Note: Evaluation in some instances involves the definition of appropriate standards, the examination of performance against those standards, an assessment of actual and expected results and the identification of relevant lessons 2. The above definitions are widely used by the development assistance community. The definitions proposed by the Policy Framework for the Government-wide Monitoring and Evaluation System 3 also broadly accord with the above definitions. Evaluation is the determination of merit or shortcoming. To make the judgement one needs a standard of what is regarded as meritorious to compare with. Evaluation is thus a process of comparison to a standard. For instance, the statement a high quality service has been delivered that met the needs of clients and improved their circumstances is an evaluation. The evaluation will be better if quality, needs and improvement in circumstances have been quantified. The emphasis in monitoring is on checking progress towards the achievement of an objective. A good monitoring system will thus give warning, early on in the implementation of a course of action, that the end goal will be reached as planned. Monitoring also involves a process of comparison because actual performance is compared with what was planned or expected. A simple example is the monitoring of the completion of the planned activities of a project 1 Organisation for Economic Cooperation and Development (OECD). Glossary of Key Terms in Evaluation and Results Based Management Ibid. 3 Presidency (Policy Coordination and Advisory Services)

12 against the target dates that have been set for each activity. Another example for routine activities like the processing of applications for social grants, is to monitor the number of applications received against the number completed per month. If 100 are received but only 90 completed and if this trend is repeated for a number of months, it means that a backlog of unprocessed applications is building up. 1.4 Importance of Monitoring and Evaluation Governments are increasingly being called upon to demonstrate results. It is expected of them to demonstrate that they are making a real difference to the lives of their people and that value for money has been delivered. Citizens are no longer solely interested in the administration of laws but also in the services that are rendered. Critically, they are more than ever interested in outcomes, like the performance of the economy in creating jobs. Similarly, the South African Government recognised that, to ensure that tangible results are achieved, the way that it monitors, evaluates and reports on its policies, projects and programmes, is crucial. In his 2004 State of the Nation address the President emphasised the importance of monitoring, evaluation and reporting in government: The government is also in the process of refining our system of Monitoring and Evaluation, to improve the performance of our system of governance and the quality of our outputs, providing an early warning system and a mechanism to respond speedily to problems, as they arise. Among other things, this will necessitate an improvement of our statistical and information base and enhancing the capacity of the Policy Coordination and Advisory Services unit. The President s statement expresses government s commitment to carry out an obligation arising from the People s Contract. Since then there has been an increased focus on M&E in South Africa. Several departments are putting in place better capacity for M&E or are developing M&E systems. The proposed Government-wide M&E System also emphasises the importance of monitoring and evaluation. 1.5 Purposes (and uses) of Monitoring and Evaluation Monitoring and evaluation is used for a variety of purposes. The purpose for which it is used determines the particular orientation of each evaluation. M&E may be used for the following main purposes: i) Management decision-making M&E systems augment managerial processes and provide evidence for decision-making. The question that should be asked is whether the quality of the M&E information provided is appropriate and how well it feeds into existing managerial processes. M&E can never replace good management practices; rather it augments and complements management. 4

13 Some examples of M&E used in this context are decisions on resource allocation, choices between competing strategies to achieve the same objective, policy decisions, and decisions on programme design and implementation. The accuracy of information and the manner in which it is presented become critical for supporting management in their decision-making processes. ii) Organisational learning This is the most challenging outcome for M&E, as it presupposes that M&E results and findings help to create learning organisations. However, translating findings into learnings challenges even the most sophisticated of organisations. M&E is also a research tool to explore what programme design, or solution to societal problems, will work best and why, and what programme design and operational processes will create the best value for money. M&E should provide the analysis and evidence to do the trade-offs between various alternative strategies. The information gathered should be translated into analytical, action-oriented reports that facilitate effective decision-making. The focus here is on causes of problems rather than the manifestation of problems. Learning has been described as a continuous dynamic process of investigation where the key elements are experience, knowledge, access and relevance. It requires a culture of inquiry and investigation, rather than one of response and reporting 4. M&E produces new knowledge. Knowledge management means capturing findings, institutionalizing learning, and organizing the wealth of information produced continually by the M&E system 5. iii) Accountability Public officials have a constitutional obligation to account to Parliament. They should be broadly accountable for how they spend public money, how they have achieved the purposes for which the money has been voted and that they have gone about their duties with a high degree of integrity. M&E provides the information, in a structured and formalised manner, which allows scrutiny of public service activities at all levels. This purpose of M&E may account for the perception that M&E is policing. Despite the concerns that many have that one should not pursue M&E only for the purpose of accountability, as it may create suspicion and a culture of fear, when dealing with public funds accountability is critically important. Accountability is governed by the Constitution and legislation such as the Public Finance Management Act, is supported by institutions such as the Auditor-General and the Public Service Commission, and failure to adhere to meeting accountability requirements is often met by sanction. 4 Kusek, J.Z and Rist, RC Ten Steps to a Results-Based Monitoring and Evaluation System. Washington, DC: The World Bank, p Kusek and Rist, p

14 Apart from the above main purposes of M&E, its findings are also used, across a broad audience, for the following 6 : i) Soliciting support for programmes If the success of a programme can be demonstrated by means of evaluation findings it is easier to garner support for the programme, for example continued or increased budgetary allocations for the programme or political support when important policy decisions affecting the programme must be made. ii) Supporting advocacy M&E results from projects and programmes generally help to make an argument for the continuation, adjustment or termination of a programme. M&E in this context provides the means for supporting or refuting arguments, clarifying issues, promoting understanding of the aims and underlying logic of policies, documenting programme implementation and thereby creating an institutional memory, and involving more people in the design and execution of the programme. Through this it plays a vital advocacy role. iii) Promoting transparency One of the most persuasive uses for M&E, if its findings are made available to a broader audience, is that it promotes transparency, and through this facilitates decision-making and accountability. M&E requires a willingness to be subjected to scrutiny, as findings may be published and made available to the public. 1.6 Content outline The chapters of this text are divided as follows: Chapter 2 explains the context of M&E in the South African Public Service. Chapter 3 introduces the idea of evaluation perspectives. Evaluation perspectives point to the main focuses of an evaluation. Chapter 4 emphasises the value basis of monitoring and evaluation and defines a range of values. Chapter 5 introduces programme evaluation and discusses a few frameworks that can be used for programme evaluation. Chapter 6 applies the concepts discussed in the foregoing chapters. 6 6 Kusek and Rist, p 130.

15 Chapter Two The Context of Monitoring and Evaluation 7

16 This Chapter deals with the following: The context of the developmental state Location of M&E in the policy process Location of M&E in the planning process The Government-wide M&E System Important institutions with a role in M&E 2.1 The context of the developmental state When state institutions or government programmes are evaluated, such evaluation should take cognisance of the type of state these institutions and programmes are located in. The Constitution envisages that the state should be a developmental state: Section 195(1)(c) provides that Public administration must be development-oriented. So, state institutions or government programmes should be designed in such a manner that they comply with this principle. The PSC expressed its views with regard to the context of the developmental state in its 2007 State of the Public Service Report: South Africa s efforts to promote growth and development are being pursued within the context of building a developmental state. Without going into a detailed discussion on the different conceptions of a developmental state, it suffices to say that such a state seeks to capably intervene and shepherd societal resources to achieve national developmental objectives, rather than simply rely on the forces of the market. What gives rise to and shapes the nature of a developmental state depends on the context and history of a country.... Against this background, many have quite correctly cautioned against any attempts to suggest that there is a prototype of a developmental state that can be constructed on the basis of what worked in other countries. What then is the specific context within which to locate a South African developmental state? The PSC believes that the Constitution provides the basis on which to understand developmentalism in South Africa given how it captures the collective will and determination of her people to create a better life for themselves 7. Monitoring and evaluation should be practised in this same context. 2.2 Location of M&E in the policy process It is important to understand where M&E fits in the policy-making and implementation cycle. A generic policy life cycle is illustrated in Figure 1 8. Since there are not many completely new problems that the state has never addressed before, the cycle probably starts with the 8 7 Public Service Commission State of the Public Service Report. Page 9. 8 Cloete, F Policy Monitoring and Evaluation. Unpublished Power Point slide.

17 review of existing policy. The stages of problem identification, determining policy objectives, examining policy options, and taking a policy decision are a complex process filtered through many layers of stakeholders. These stakeholders include political parties, civil society, legislative and executive arms of government, and government departments. Policy is further argued and explained in various documents, like discussion and policy documents. The process is invariably not as sequential or rational as depicted. Identification of options and rational evaluation of the feasibility, or the costs and benefits, of options, in any precise sense, assume perfect knowledge of what will work, which is frequently not the case. Policy options emerge through political debate and the best policies through taking a considered decision and making adjustments when the effect of a policy is seen in practice. As soon as a policy decision has been taken, government departments initiate the processes of designing a programme that can achieve the policy objectives, detailed planning of the programme and implementation. To ensure that implementation proceed as planned and that the envisaged objectives are achieved, the programme is monitored and evaluated. Depending on the results achieved by the programme, the initial policy decision, or aspects of the design, implementation and resource allocation to the programme, may be reviewed. Figure 1: The Policy Life Cycle Problem Review Policy Objectives Evaluation Policy Options Monitoring Implementation Feasibility of options Planning Policy Decision Figure 1. The policy life cycle From the depiction of the cycle in Figure 1 it can be seen that the evaluation of the success of policy and the reasons for success or failure, are critical parts of the process. This evaluation is not necessarily a formal, technical evaluation but one that is intricately part of administrative 9

18 and political processes where the judgements and power of key decision-makers play the primary role. M&E mediates this by producing valid evidence for policy decisions, thereby ensuring greater objectivity. Since public policy is a set of statements that determine what actions government will take, what effects those actions will have on social conditions, and how those actions can be altered if they produce undesirable outcomes 9, policy evaluation is also an inherent part of M&E. 2.3 Location of M&E in the planning process Having located M&E in the policy cycle, it is also necessary to explain where it fits into the more formal planning and implementation processes of government departments. This process is illustrated by Figure Figure 2: Planning and Review Cycle 1. Preparation of Performance Plans 5 Year Strategic Plan 4. Review Annual review Medium Term Budget Annual Performance Plan Specially Commissioned Evaluations Third Quarter Report 2. Monitoring Performance plans for units and individuals Performance Information Data Base Monthly and quarterly reports Figure 2. Planning and Review Cycle In the national sphere of government each department must produce a five year strategic plan, which is aligned with government strategic direction as expressed in the Medium Term Strategic Framework and the Government Programme of Action. The process starts with each new electoral cycle when a new government produces a new programme of action. The same happens at provincial level where strategic plans must be aligned with the provincial government programme of action, but also to national plans Fox, W, Schwella, E and Wissink, H. Public Management. Juta, Kenwyn p Ibid.

19 This is especially true for concurrent functions where the national government produces plans for the whole sector that guide planning and implementation at the provincial level. At a provincial level, departmental strategic plans should also be aligned with the Provincial Growth and Development Strategies 11. Plans must further be aligned with local Integrated Development Plans 12. Ideally, formal plans produced by government departments must be aligned across all spheres of government 13. Based on its strategic plan each department prepares its budget (called Estimates of Expenditure/ Medium Term Expenditure Framework 14 ), which is submitted to the treasury and eventually approved by Parliament or the provincial legislature. When approved by Parliament or the provincial legislature the budget becomes law (the Appropriation Act for the year) and is a department s mandate to spend money on the purposes for which it was voted. Based on its strategic plan and the medium term budget, a department must also prepare an annual performance plan. These plans the strategic plan, the budget and the annual performance plan contain objectives, outputs, indicators and targets. A department s annual performance plan is broken down to plans for each component in the organisation. These plans are then implemented and monitoring should start immediately. The monitoring measures progress against the objectives, outputs, indicators and targets in the plans and takes the form of monthly and quarterly reports. Managers supplement quarterly monitoring with evaluation of success, analysis of the reasons for success or shortcoming, and action plans for improving performance. Managers own evaluations can further be supplemented by specially commissioned evaluations by experts. This process culminates in an annual review of performance, which feeds into a new planning cycle for a following financial year. 2.4 The Government-wide M&E System Cabinet has mandated the Governance and Administration Cluster of the Forum of South Africa s Directors-General to construct an overarching Government-wide Monitoring and Evaluation System. The system is envisaged to function in two ways 15 : It should provide an integrated, encompassing framework of M&E principles, practices and standards to be used throughout Government; and 11 A comprehensive development plan for the province, showing how various stakeholders, including government on national and provincial level, and the private sector, will contribute to the development of the province. 12 A comprehensive development plan for the local area, showing how various stakeholders, including the local government, the private sector and government departments in other spheres of government, will contribute to the development of the local area. 13 See the background section (Linking Strategic Planning to the Electoral Cycle) in: National Treasury. Framework and Templates for provincial departments for the preparation of Strategic and Performance Plans for , and Annual Performance Plans for the 2005 financial year. 16 August In South Africa the published budget contains estimates for three years, or medium term. 15 Presidency Policy Framework for the Government-wide Monitoring and Evaluation System. Page 5. 11

20 it should also function as an apex-level information system which draws from the component systems in the framework to deliver useful M&E products for its users. It is therefore a derived system that extract information from other systems throughout all spheres of government. It will therefore be very reliant on a minimum level of standardisation throughout government, as well as the quality of information from those systems. The GWM&E system will produce the following outputs 16 : Improved quality of performance information and analysis at programme level within departments and municipalities (inputs, outputs and outcomes). Improved monitoring and evaluation of outcomes and impact across the whole of government through, eg Government Programme of Action bi-monthly Report, Annual Country Progress Report based on the national indicators, etc. Sectoral and thematic evaluation reports. Improved monitoring and evaluation of provincial outcomes and impact in relation to Provincial Growth and Development Plans. Projects to improve M&E performance in selected institutions across government. Capacity building initiatives to build capacity for M&E and foster a culture of governance and decision-making which responds to M&E findings. Government draws from three data terrains for M&E purposes, each of which will be subject to a dedicated framework describing what is required for them to be fully functional. The three terrains are depicted in Figure Figure 3: Policy Frameworks of the Government-wide Monitoring and Evaluating System Government-wide Monitoring and Evaluation Framework Programme Performance Information Evaluations Evaluations Framework Social, Economic and Demographic Statistics Statistics and Surveys Framework Framework for Managing Programme Performance Information Presidency Policy Framework for the Government-wide Monitoring and Evaluation System. Page Presidency Policy Framework for the Government-wide Monitoring and Evaluation System. Page 7.

21 Figure 3: Policy Frameworks of the Government-wide Monitoring and Evaluation System Two of these frameworks have already been issued, namely the Framework for Managing Programme Performance Information, issued by the National Treasury in 2007, and the South African Statistical Quality Assessment Framework (SASQAF) (First edition) issued by Statistics South Africa, also in As currently conceptualised, the Government-wide System relies on systems in departments in all spheres of government to provide the information from which the performance of the whole of government can be judged. The Policy Coordination and Advisory Services in the Presidency have already produced important whole of government performance reports, for example, the Towards a Ten Year Review in , and the Development Indicators Mid-term Review in , based on data supplied by departments, that is, from systems that feed into the Government-wide system. Work done by the National Treasury to improve the quality and also standardise performance indicators, has already had an impact on the quality of performance information. The efforts of the National Treasury were complemented by the Auditor-General who, from the 2005/06 financial year, started with a process of auditing the quality of performance indicators, which will eventually lead to the auditing of the quality of the performance information itself. 2.5 Important institutions with a role in M&E A document of this nature cannot exhaustively cover the complex machinery of government 20, but since many institutions have a role to play in M&E and since these roles often overlap, a short explanation of the roles of key institutions is given here. This short section only gives a few signposts, and can therefore not be complete in the sense of covering all the institutions that have an M&E role. Specifically, institutions on the local government level are ignored. Explaining the roles of these institutions may be made easier by grouping them into the following categories: Departments at the centre of government 21 on the national level. Departments at the centre of government on the provincial level. Line departments. Constitutional institutions. 18 The Presidency, Policy Coordination and Advisory Services (PCAS). Towards a Ten Year Review: Synthesis report on implementation of government programmes. October The Presidency. Development Indicators Mid-Term Review A quick reference to the machinery of government is: DPSA. The Machinery of Government: Structure and Functions of Government. May The phrase at the centre of government is used here to denote departments which regulate, give direction to or render services to other departments (rather than the public). 13

22 2.5.1 Departments at the centre of government on the national level The Presidency The Presidency supports the President to give central direction to government policy. In this regard the Presidency is a key role player in compiling the Medium Term Strategic Framework and the Government Programme of Action. The Presidency then monitors the implementation of key government priorities. Specifically, the Presidency compiles bi-annual progress reports on the implementation of Government s Programme of Action. It also monitors the performance of South Africa against key development indicators. The Presidency was, for example, responsible for the Ten Year Review 22 and the Development Indicators Mid-term Review 23. For these reports the Presidency is dependent on data that it draws from several government departments and it is therefore essential that the M&E systems in these departments can absolutely be relied upon. National Treasury The National Treasury supports the Minister of Finance to determine fiscal policy. As such it must closely monitor a range of economic indicators. The National Treasury further compiles the national budget and develops and implements financial management policy. Since money is, through the budget, allocated by Parliament to achieve specific strategic objectives, and indicators and targets are set in the Estimates of Expenditure to measure the attainment of those objectives, the National Treasury plays a key role to monitor performance against objectives. This is done by means of quarterly reports that must be submitted to the National Treasury. The National Treasury also evaluates whether the expenditure has achieved value for money. These evaluations are published in key documents like the Budget Review 24, the Provincial Budgets and Expenditure Review 25 and the Local Government Budgets and Expenditure Review 26. Department of Public Service and Administration (DPSA) The DPSA is responsible for the macro-organisation of the Public Service and for the development of policy on the functioning of the Public Service. It is further responsible for Human Resource Management policy, the determination of conditions of service for the Public Service, the development of policy, regulations, norms and standards for Information Management and the use of Information Technology in the Public Service, and generally for the promotion of a Public Service that conforms to all the values governing public administration listed in Section 195 of the Constitution. As such the DPSA must monitor and evaluate the performance of the Public Service, especially from the perspective of sound Human Resource Management The Presidency, Policy Coordination and Advisory Services (PCAS). Towards a Ten Year Review: Synthesis report on implementation of government programmes. October The Presidency. Development Indicators Mid-Term Review National Treasury. Budget Review February National Treasury. Provincial Budgets and Expenditure Review 2003/ /10. September National Treasury. Local Government Budgets and Expenditure Review 2001/ /08. October 2006.

23 Department of Provincial and Local Government (DPLG) The DPLG develops policy on the structure and functioning of provincial and local government and as such monitors and evaluates the performance of provincial and local government. Since local government is a critical institution for the delivery of basic services, the DPLG s role in monitoring, and evaluating, the financial health and the service delivery of local government, is a very important role. Statistics South Africa (Stats SA) Stats SA manages the national statistics system that collects, analyses and publishes a range of demographic, social and economic statistics. It also collects statistics on a set of key development indicators. Without a sound base of reliable statistics, planning of government services, and M&E, at a level of sophistication that is required of government, will not have been possible. South African Management Development Institute (SAMDI) SAMDI provides or commissions training in M&E for the Public Service Departments at the centre of government at the provincial level At provincial level the key departments with a role in M&E are the offices of the premier and provincial treasuries. From an M&E perspective, key strategic objectives for the province are set in the Provincial Growth and Development Strategy and the Provincial Government Programme of Action. Offices of the premier play a key role in setting strategic direction for the province and monitoring and evaluating the performance of the provincial government departments on the delivery of the growth and development strategy and other provincial priorities Line departments Line departments implement government policy in their specific functional areas. As part of this role they must monitor and evaluate the implementation of policy, the impact of policy, as well as the level and quality of service delivery. A critical role is played by the national policy departments for concurrent functions because they must develop policy as well as norms and standards for M&E systems that will be used throughout the sector. This includes a standard set of performance indicators for the sector so that performance can be compared across the sector Constitutional institutions Though the functions of the constitutional institutions may often overlap with the above categories of institutions, their role differ in the sense that they do monitoring and evaluation independently from government and don t report to the Executive, but to Parliament. 15

24 They therefore have a special role to play to protect the values and principles of our democracy. Because they are independent, they may arrive at different conclusions about the performance of government and the Public Service, and as such may bring different analyses and insights to bear on public policy. Public Service Commission (PSC) The PSC has the constitutionally prescribed function to promote the values and principles governing public administration listed in section 195 of the Constitution, in the Public Service. It must further monitor and evaluate the organisation and administration of the Public Service and can propose measures to improve the performance of the Public Service. It must also provide to Parliament an evaluation of the extent to which the values and principles governing public administration have been complied with in the Public Service. Based on these functions the PSC aims to establish itself as a leader in monitoring and evaluation of the performance of the Public Service. It is in the context of these functions that the PSC is publishing this document. Auditor-General The Auditor-General s role is to audit the accounts and financial statements of national and provincial departments as well as municipalities and any other government institution or accounting entity. These audits include financial audits to certify that the institution s financial statements fairly represents the financial position of the institution and regularity audits to certify that the institution has complied with all relevant regulations and prescripts. Important from an M&E point of view is that the Auditor-General also does performance audits to test whether money has been spent economically, efficiently and effectively by the audited entity. The Auditor-General has also started to do audits and express an opinion on the quality of performance indicators that departments publish in their strategic plans and in the Estimates of National Expenditure. The aim is to, in future, also audit performance information, so that the Auditor-General can express an opinion on the veracity of such performance information just as it is expressing opinions on the veracity of financial information. It will be a big forward step for the practice of M&E if secondary users of performance information could have confidence in such information, based on the knowledge that the accuracy of the information has been tested by the Auditor-General. Human Rights Commission The Human Rights Commission was formed to promote a culture of human rights, respect for, and protection of these rights as enshrined in the Bill of Rights, and to monitor and evaluate the extent to which human rights are observed in South Africa. Since many of the rights are socioeconomic rights, the Human Rights Commission also plays a role in monitoring and evaluating service delivery by government 27. In cases where the rights of individuals and communities have been violated, the Commission has the power to secure appropriate corrective action An important report in this regard is Human Rights Commission th Economic and Social Rights Report.

25 Chapter Three Evaluation Perspectives 17

26 This Chapter deals with the following: The idea of evaluation perspectives The Balanced Scorecard of Kaplan and Nortan Programme performance perspective Financial perspective Governance perspective Human Resource Management perspective Ethics perspective The perspectives adopted by National Treasury Guidelines 3.1 The idea of evaluation perspectives The subject of an evaluation (the topic, the entity to be evaluated) may be the Public Service, a system, policy, programme, several programmes, a service, project, a department or unit within a department, a process or practice. The subject of an evaluation may also be the whole of government or the country. These entities are complex, or multi-dimensional. For analytical purposes, some framework is needed to identify the dimensions that will be focused on. All these entities are intended to do something or to result into something. Thus, their performance can be evaluated. Evaluation perspectives point to the main focuses of an evaluation. By simply asking a few questions one can see that the performance of the Public Service can be viewed from many different perspectives, for example: How was the money voted by Parliament spent? Was there any wastage? Was value for money obtained? Was government s policy objectives achieved? Is the Public Service corrupt? Are people treated fairly and courteously in their interaction with the Public Service? When complex, multi-dimensional subjects are evaluated the outcome will depend largely on the perspective one adopts, or what dimensions of the complex reality are emphasised, or what questions are asked. The type of evaluation and the methodologies used, and consequently the outcomes of the evaluation, depend on the questions asked. The central idea behind evaluating performance from different perspectives is to use a balanced set of perspectives for the evaluation. For example, evaluation of the achievement of policy outcomes (programme performance perspective) may be balanced by evaluation 18

27 from a financial perspective (to evaluate for instance whether value for money has been obtained). An organisation may be very efficient in delivering a specific output (business process perspective), but not in adapting to changes in the policy environment or the specific needs of citizens (learning and growth perspective). During particular stages of implementation of a programme, different perspectives may be emphasised. When a programme is still in the design phase or when it comes up for policy review, policy analysis and review are appropriate. Outside periods of policy review, issues of implementation are more appropriate, to make sure that programmes do not fail because of poor implementation. Evaluation of the performance of the Public Service from a particular perspective employs analytical frameworks, models, theories and methodologies unique to the particular perspective. For instance, financial analysis makes use of financial information prepared according to specific accounting practice and the information is analysed using accepted methods and models. Evaluation from a human resource management perspective involves specific frameworks regarding the objectives of human resource management and the practices employed to achieve those objectives. Similarly, the evaluation of service delivery or programme performance also employs specific frameworks. Some frameworks for the evaluation of programme performance are introduced in Chapter 5. This document does not advocate a specific set of perspectives, but in this section examples of possible perspectives are discussed to illustrate how perspectives can aid in framing an evaluation (deciding what to evaluate, the purpose of the evaluation, and the main evaluation questions). Since the idea of using different perspectives can be credited to Kaplan and Nortan 28, their four perspectives are discussed first (as illustration, not prescription). Important perspectives in the context of the Public Service are discussed next. Lastly, we turn to perspectives prescribed/ suggested by National Treasury guidelines to show that the idea of perspectives has already found its way into Public Service prescripts. 3.2 The Balanced Scorecard of Kaplan and Nortan Kaplan and Nortan have proposed the following four perspectives for evaluating the performance of an organisation (though their focus was on the private sector): Financial Perspective: To succeed financially, how should we appear to our shareholders? Customer Perspective: To achieve our vision, how should we appear to our customers? Learning and Growth Perspective: To achieve our vision, how will we sustain our ability to change and improve, and adapt to changes in the environment and new challenges? Internal Business Process Perspective: To satisfy our shareholders and customers, what business processes must we excel at? What are the unique competencies the organisation should have? 28 Kaplan, RS and Nortan, DP, The Balanced Scorecard. Harvard Business School Press, Boston, Massachusetts. 19

28 A business may not get early warning about threats to its sustainability if it is only evaluated from a (especially short term) financial perspective. Realising this, Kaplan and Norton developed the alternative perspectives from which the performance of a business may be evaluated, to give a more balanced view of such performance. 3.3 Programme performance perspective A programme is a set of government activities that deliver the products of government. These products are complex outcomes and include governance, justice, safety and security, development impetus, social change and services. Evaluation from a programme performance perspective will try to answer questions such as whether government objectives have been achieved and whether it could have been achieved better by designing the programme differently or implementing it better. Effective monitoring and evaluation of government programmes requires careful analysis of the key factors that are relevant to the successful delivery of the programme, and of how these relate to each other. Different approaches are available to facilitate such analyses. Evaluating programme performance is discussed in Chapter Financial perspective Monitoring and evaluation from a financial perspective happens through monthly and annual financial statements. Financial statements try to answer the following questions: Was money spent as appropriated, has the income that accrued to government been collected, were assets protected, can the department meet its liabilities and has the department adhered to sound financial controls? These financial reports currently primarily give managers updates on progress with expenditure as measured against budget. Such statements are prepared in terms of Generally Recognised Accounting Practice as prescribed in terms of the PFMA. Annual financial statements are also audited by the Auditor-General so that a high degree of confidence could be attached to financial figures. Since financial accounting answers very basic questions some departments are trying to introduce management accounting with tasks of analysing and interpreting financial information, costing services, advising managers on the financial implications of strategic decisions, advising on choosing between alternative strategies, and directing attention to and helping managers to solve problems. So, as with other types of M&E, the process begins with monitoring and answering basic, pre-set questions and as more and more questions are asked, the more penetrating the evaluation becomes. 20

29 3.5 Governance perspective The PSC views good governance as compliance with all the values listed in section 195 of the Constitution 29. It has been argued, for instance by Cloete 30, that A coherent good governance measurement programme should be developed as a matter of urgency as an integral part of a more encompassing M&E programme in South Africa. Governance has been conceptualised by Olowu and Sako 31 as a system of values, policies and institutions by which a society manages its economic, political and social affairs through interaction within and among the state, civil society and private sector. Government s Ten Year Review 32 used indicators grouped under the following categories to measure governance: Voice and accountability. Political instability and violence. Government effectiveness. Regulatory quality. Rule of law. Ethics. It is clear that good governance is a specific perspective and that the monitoring and evaluation of the performance of South Africa, the Government or the Public Service from this perspective requires unique approaches, methodologies, indicators and data sources. 3.6 Human Resource Management (HRM) perspective Similar to all the other perspectives discussed in this section, the Human Resource Management perspective also requires the application of unique approaches to M&E. An evaluation from a HRM perspective requires evaluation of both whether HRM objectives have been achieved; and good human resource management practice is applied in the Public Service. 29 See State of the Public Service Report for several years. 30 Cloete, F. Measuring good governance in South Africa. November (Unpublished article.) 31 Olowu, D and Sako, S. Better Governance and public Policy: Capacity Building and Democratic Renewal in Africa. Kumarian Press, Bloomfield, USA As quoted by Cloete, F. Measuring good governance in South Africa. November The Presidency, Policy Coordination and Advisory Services (PCAS). Towards a Ten Year Review: Synthesis report on implementation of government programmes. October