Indicators and measurement of government policies

1.2.3. Public Value & Policy Analysis and Evaluation Day 11 Indicators and measurement of government policies Compiling indicators

Trainer Introduction Mr Jani Heikkinen Head of Division, Statistics Finland Goal of this week Increased understanding of formulation and usage of government policy indicators

Agenda Setting and usage of indicators at different levels of government: policy level, agency level and operational level The major challenges and pitfalls of indicators and their measurement Overview of usage of indicators across OECD countries Examples of setting and usage of indicators (Finnish and/or OECD country cases) Discussion and Q&A Summary of main learning points

Setting and usage of indicators Major points to start with: What do you want to measure? On what theory your measurement bases on? How goals set and measured by indicators correspond to your own activities?

Why? Need Policy goal Supply goals Resource need Measure Measure Measure Measure Satisfaction Policy achievement Supply Resource use 5

Selecting classification for indicators Setting up indicators bases on selected classification system Focus Comparability Classification bases on theory Simplicity when forming measurement groups Relevance Classifications (and theories) are connected Shows causality of measures and actions

Linking classifications Public Core Values Production Chain Areas of Activities Management Procedures and Practices Budgeting Procedures and Practices Human Resource Management Procedures and Practices Regulatory Management Procedures and Practices E-government Procedures and Practices Central Level of Government State Level of Government Local Level of Government Type/Sector of Production General Public Services Police Services Education Services Health Services Social Services Mi cro-level Meso-level Whol e of Government-level Procurement Procedures and Practices Internal Services

Objective vs. Subjective GDP vs. opinion Positive vs. Normative Number of treatments vs. no: of wrong diagnosis Economical vs. Social Income vs. people widowed Additive vs. Composite Productivity vs. Actionable governance indicator Survey vs. Poll Selection of indicator families Budget practises vs. Consumer Survey

Operational Decision Making and Performance Management in Public Finance Selecting measurement level Policy goal Policy Ministry A Minsitry B Agency A1 Agency A2 Agency B1 Agency B2 Activity A1.1 Activity A1.2 Activity A1.3 Activity A2.1 Activity B1.1 Activity B2.1 ActivityB2.1 9

Selecting measurement level Wellfare policy: Life expectancy Ministry of health: Health policy: Death due to health issues Ministry of Transport : Transport policy: People killed in road accidents Hospitals (Local government): Number of heart related treatments Institute for Health : Research Transport Safety Agency: safety measures Transport Agency: Total Capacity Treatmets of accute heart attack Treatments of Aterosklerosis Number of preventive consultations New research on causes of hearth diseases Number of inspections to wehicles Km s of roads with speed cameras Km s of roads with safety barriers 10

Example of theory and measurement Production theory Input Process Output Outcome Measured by: Labour Surgery Treatment Health

Example of theory and measurement Theory of good governance Rule of law Integrity Transparency Predictable governance Equality

Example of theory and measurement Measures of good governance What share of total procurement the procurement law covers Experienced corruption in procurement Share of openly announced procurement of total procurement Expressed Trust in government Experienced barriers for operators when entering procurement markets

Selecting reporting systems Financial statement Annual report Income statement Balance sheet etc... Balanced Scorecard Performance Prism Performance Pyramid System Objective Matrix

Outcome targets Operational performance targets Operational efficiency -economy -productivity -profitability -cost-equivalence Policy effectiveness Outputs and quality management -goods and services -service capacity and quality Policy effectiveness: outcomes How operations and finances have affected policy effectiveness Operational results: outputs (which can be influenced through management) Human resource management

T O T A L Q U A L IT Y A ffect Elem ents of q ua lity nfluence M eeting the requirem ents Q uality of input Q uality of produ ction Q uality of ou tp ut N eeds Q uality of s ys tem O b jectives In pu t P rod uctio n O utput O u tco m e Factors of prod uction C osts C apacity P erfo rm ance Im pa cts P rofits R ea l R E A L A S - S E T S Ec ono m ic efficiency Pro du c tivity Effe ctivenes s PR O F IT A B IL IT Y

The Measurement pyramid Policy goal indicators Activity/service indicators

Challenges and pitfalls You get what you measure!

Challenges and pitfalls No worries when indicators are: Positive, Objective, Economical, Additive Not too closely tied with incentives Examples: GDP Income

Great risk of going wrong with indicators when: Part of reward system Only partial coverage Subjective, Normative, Social, Composite, Poll Describing instead of numeric Examples: Challenges and pitfalls productivity driven bonus system wellbeing

Good requirements for measures Simple and understandable communication Theory focus Cheap more with less preventing pitfalls Multi usage all levels benefit Coverage all relevant inn Controlled by other data no negative side effect Open and transparent trust 21

Usual story behind indicator development Policy /indicator assessment Policy (sector) goals Developing domestic indicators Reporting Seeking international comparison Data collection Harmonisa tion (eg. By OECD)

Comparative policy indicators at Demand from Goals for Countries governments Interest groups Country comparison Policy evaluation Policy specific targets Multiple providers international level Survey general statistics poll 23

Government at a Glance, OECD http://www.oecd.org/document/33/0,3746,en_2649_33735_43714657_1_1_1_1,00.html Health at a Glance, OECD http://www.oecd.org/document/19/0,3343,en_2649_33929_46460563_1_1_1_37407,00.html Education at a Glance, OECD http://www.oecd.org/document/52/0,3746,en_2649_39263238_45897844_1_1_1_1,00.html Society at a Glance, OECD http://www.oecd.org/document/24/0,3746,en_2649_34637_2671576_1_1_1_1,00.html Sustainable Governance Indicators (SGI), OECD, Bertelsmann-Stiftung Examples of available Policy Indicators from OECD countries http://www.sgi-network.org/index.php

Actionable Governance Indicators (AIG), World Bank https://www.agidata.org/main/src.ashx World Development Indicators (WDI), World Bank http://data.worldbank.org/indicator Millennium Development Goals Indicators (MDG), UN Examples of available Policy Indicators from OECD countries http://mdgs.un.org/unsd/mdg/default.aspx Economic indicators: UN, IMF, World Bank, OECD, Eurostat, National statistical agencies...

1,0 0,9 0,8 0,7 0,6 0,5 0,4 Government at a Glance Use of performance budgeting system at the central level of government (2007) 0,3 0,2 0,1 0,0 FINLAND AUSTRALIA KOREA NETHERLANDS NORWAY POLAND UNITED STATES CANADA UNITED KINGDOM SWEDEN TURKEY SLOVAK REPUBLIC JAPAN NEW ZEALAND AUSTRIA DENMARK IRELAND MEXICO OECD30 GREECE LUXEMBOURG FRANCE ITALY ICELAND SWITZERLAND PORTUGAL GERMANY HUNGARY SPAIN BELGIUM CEZCH REPUBLIC Government at a Glance 2009 - OECD 2009 - ISBN 9789264061644 26

Education at a glance 1,00 0,90 0,80 0,70 0,60 0,50 0,40 0,30 0,20 0,10 0,00 Proportion of adults reporting good health, by level of education Below upper secondary education Upper secondary education Tertiary education OECD 27

Health at a Glance 2007 (or latest year available) Source: OECD Health Data 2009, OECD (http://www.oecd.org/health/healthdata).

Society at a Glance Considerable differences between countries regarding life satisfaction Average points of life satisfaction on an 11-step ladder from 0-10, 2006 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 1 0 TUR ITA SVK HUN PRT KOR POL IRL GRC CZE JPN DEU MEX LUX ISL GBR FRA AUT ESP USA SWE BEL CAN AUS NZL CHE NOR NLD FIN DNK 0 Society at a Glance 2009: OECD Social Indicators - OECD 2009 - ISBN 9789264049383 29

Use of indicators in countries Thumb rule If you have it in international level you have it in domestic level Every country tries to learn from each other Usage varies Usage of performance budgeting Policy relevance Institutional differences Culture 30

Developing path of indicators by COFOG* Education Health Social protection Economic affaits General public services Recreation,culture and religion Housing and community amenities Environmental protection Defence *UN, Classification of the Functions of Government 31

CASE: Statistics Finland: Measures STATISTICS FINLAND IN BUDGET, Operative Goals 2010 Nature of measure Operative efficiency Change of productivity, % 2 Productivity Share of fixed cost out of total costs, % 74 Input Share of costs of producing market goods out of sales, % 100 Efficiency Outputs and quality control Number of publications 665 Output Late publications, at most 35 Output quality Production costs per publication, t 75,9 Efficiency Publication speed, weeks annual statistics 38 Service quality quarterly statistics 8,3 Service quality monthly statistics 4,8 Service quality Number of free database tables, 1 100 Output quality Share of critical feedback out of total feedback, % 28 Service quality

Planning Budget: government parliament Performance agreement: Agency government (minisitry) Performance agreement : Department Agency Reporting Annual report by levels: CASE: Statistics Finland: Use Government (in budget) Agency Department Finnish State Internet Reporting (NETRA) 33

CASE: Statistics Finland: Use Compilation bases on Own performance reporting system Activity based follow up of expenditures/costs and income Data more in detail at detailed level (agency level vs. Office level) Measurement is present in every level of organisation Person>team>office>department>agency (>ministry>government) 34

CASE: Statistics Finland: Method applied by central government agencies Agency Government Departme nt Ministry Office Team Agency

Key points of today Multiple measurement applied by various countries ability for comparison reached in certain fields same goal variation in execution Measurement needs good foundations theory, focus avoids 36