Assessing the Employment and Social Impacts of Selected Strategic Commission Policies

Transcription

1 Assessing the Employment and Social Impacts of Selected Strategic Commission Policies Database fiches January 2009 DG Employment, Social Affairs and Equal Opportunities of the European Commission ECORYS Nederland BV IDEA Consult NV

2 ECORYS Nederland BV P.O. Box AD Rotterdam Watermanweg GG Rotterdam The Netherlands T +31 (0) F +31 (0) E netherlands@ecorys.com W Registration no Fortis Bank no IBAN NL14 FTSB BIC FTSBNL2R VAT no. NL B.08

3 1 Overview of methods to measure social impacts 1.1 Introduction The assessment of social impacts of strategic policies outside the social domain crucially depends on the availability of techniques to assess these impacts. Hence, part of this project was to construct a database that provides an overview of existing techniques that are used both inside and outside the social domain to assess social impacts of policy measures. In this chapter 1 we provide a brief description of the approach followed to collect relevant impact assessment techniques. Next an overview is given of the database content. The database is developed in Excel and its content is available in chapter 2 of this document. A fiche is provided for each technique, whereby we specifically focus on the assessment of social impacts. 1.2 Approach to construct database In order to collect techniques for the database the following steps were made. In this section we shortly describe the following issues: The criteria for selection of techniques; Search methods that were used to collect the techniques; Selection criteria For the database the following selection criteria were used: Quantitative as well as qualitative techniques Being able to assess (some) social impacts Within the category of models: - General as well as specific models - Global and local models - Sectoral and general models 1 The description of the approach followed to construct the database and the description of the database content are also reported in the final report of the study (chapter 2). 3

4 Search method employed: For the collection of techniques the following sources were checked: As a start the IA TOOLS 2 was consulted. IA TOOLS is an online platform that provides Commission policy actors and impact assessment practitioners throughout Europe with a repository of guidance, information and best practices for the impact assessment of new policies and legislative measures. Besides IA TOOLS also provides a model inventory that helps to determine whether the impacts of a certain policy proposal can be assessed and quantified using existing models. The PSIA 3 -website of the World Bank. The website presents information on a series of tools and methods available for the analysis of poverty and social impacts of reforms. It presents summary information on the tools, drawing in particular on the Toolkit for Evaluating the poverty and distributional impact of economic policies and Tools for institutional, Political and social analysis (TIPS). The DIADEM project on disseminating on regulatory impact assessment. 4 DIADEM aims to document the differing modes and degrees of implementation of assessment procedures in EU member states and at the EU level. The MEANS collection Evaluating socio-economic programmes, of the European Commission. This publication gives wider access to the lessons of the MEANS programme which the EC introduced with a view to improving and promoting evaluation methods for Structural Policies. Finally, we consulted the ECORYS experts on transport, internal market, trade and energy. The experts are all involved in this study. Among other an in-house document was used that provides an overview of impact assessment methodologies. 5 Next, the links to specific model websites were followed and these websites were examined generally and specifically for information on the technique or model and links to documentation (working papers, articles). Non-EU funded models (that aren t described on the IATOOLS website) websites were directly visited. Articles on model applications or technical papers on models that were found through a models website or by searching the internet, were then analyzed specifically with regard to the measurement of social issues. Valid information for the database was extracted from the paper or article. Promising references or links to other sources of information on a model found in an article or paper were also investigated. For a limited number of the most promising models a web-based interview was sent to the model builders. 2 IA TOOLS website 3 Poverty and Social Impact Analysis ~theSitePK:490130,00.html 4 See European network for Better Regulation: 5 ECORYS, Structured Overview Impact Assessment Methodologies, ECORYS Research Programme, Project CD17500 Impact Assessments and Industrial Competitiveness, Draft version of 9 April

5 1.3 General description of the database content Table 2.1 provides a complete overview of the content of the database of techniques. Table 1.1 Overview of the content of the database Name technique General description of the technique Selection criteria Specific aspects related to social impact assessment References Quantitative or qualitative Model type (only for quantitative techniques) Sectoral or general Local or global Stage in the evaluation or IA Substage in the evaluation or IA Regions Assumptions Specific limitations Main policy field (e.g. transport, energy) Data requirements (only for quantitative techniques) Development and run time (only for quantitative techniques) Description General outputs Social impacts: - Employment - Job quality - Standards and rights related to job quality - Social inclusion and protection - Non-discrimination and equality of treatment and opportunities - Access to and effects on social protection, health and educational services - Public health and safety External link Complementary techniques Articles and references found 5

6 Some of the elements indicated in Table 1.1 require some extra explanation: Stage in the evaluation or impact assessment: the different techniques can be ordered according to the stage in the evaluation or impact assessment in which they can be used. Four key stages can be distinguished: Structuring (identify expected effects or defining observation instruments) Observing (defining field of observation or collecting data) Analysing (comparing observations or estimating effects) Judging (judging effects per criterion or formulating a synthetic judgement) Some techniques can be used in several stages. For each technique, the most relevant stage (or stages) is/are indicated. Specific limitations: Each technique has its advantages and drawbacks. The general advantages and limitations of qualitative and quantitative techniques are described in chapter 4 of the final report and summarised below. For some quantitative techniques additional limitations are indicated in the database fiches. In general quantitative techniques have the following advantages: They permit generalizations Other researchers can independently replicate and verify the results They can establish a sense of causality The results are seen as objective and impartial At the same time, some drawbacks of quantitative techniques are: It is difficult to capture context specific issues with quantitative techniques They are designed externally, reflecting assumptions determined by the researcher developing or running the model Generally speaking, qualitative techniques are capable of assessing the impact on social issues within a relatively short time-period. When quantitative data is unavailable, assessment of these social impacts is possible by using qualitative techniques. Moreover, these techniques can take fully account of context-specific issues. However, qualitative approaches do suffer from a number of important drawbacks: The individuals or groups being studied are usually small in number and/or have not been randomly selected, making it highly problematic (though not impossible) to draw generalizations about the wider population. Because groups are often selected idiosyncratically, e.g. on the basis of a judgment call by the lead investigator or on the recommendation of other participants, it is difficult to replicate, and thus independently verify, the results. Qualitative analysis often involves interpretative judgments on the part of the researcher, and two researchers looking at the same data may arrive at different interpretations. Because of an inability to control for other mitigating factors or to establish the counterfactual, it is hard to make compelling claims regarding causality. Development and run time: A priori it is not possible to indicate which techniques are more cost efficient. A model can be quickly run if all data is available and up to date. If this is not the case or if the model needs to be adapted high costs can be incurred and more time will be needed. The cost price of executing qualitative approaches can also vary a lot (e.g. depending on size of survey, number of interviews, etc.). In the database fiches an indication of the development and run time is only specified for the quantitative techniques. 6

7 Model type: the following distinction is made within the group of quantitative techniques 6 Computable General Equilibrium models Partial Equilibrium models Sectoral models Macro-econometric models Micro-models Social impacts: for each technique it is indicated whether social impacts can be assessed. This is done for each of the 7 domains (i.e. employment, job quality, standard and rights related to job quality, social inclusion, non-discrimination, access to and effects on social protection/health/education, public health and safety). 1.4 Overview of techniques to assess social impacts In total 58 techniques are described in the database (see next chapter). The table below classifies these techniques according to the stage in the evaluation or impact assessment in which they can be used (i.e. structuring, observing, analysing or judging). The fiches in the next chapter are ranked according to the table below. Table 1.2 Overview of techniques to assess social impacts. Quantitative or Model type (only for Page Stage in evaluation or IA qualitative technique quantitative techn.) Name technique number Structuring Qualitative Causal chain analysis 10 Qualitative Concept mapping 12 Qualitative Impact matrix (Leopold matrix) 14 Qualitative Individual interview 26 Qualitative Logical framework 16 Qualitative Objectives diagram and impact diagram Qualitative Problem diagram 20 Qualitative Stakeholder analysis 32 Qualitative SWOT analysis 22 Observing Qualitative Focus group 24 Qualitative Individual interview 26 Qualitative Questionnaire survey 28 Qualitative Social Capital Assessment Tool 30 Qualitative Stakeholder analysis 32 Analysing data Quantitative Computable General CGE 34 Equilibrium Augmented CGE with 36 representative households PRSP model 38 EDGE 40 EDIP 43 GEM-CCGT 45 IMMPA 48 6 For more information on this typology we refer to chapter 4 in the final report. 7

8 Analysing data Quantitative Computable General LINKAGE 50 Equilibrium OECDTAX 52 PACE 54 TEQUILA 55 Worldscan 57 Quantitative Macro-econometric E3ME 59 model Input-Output model 63 NEMESIS 66 PAMS 68 QUEST II 70 RIOT 72 SASI 74 Quantitative Micro-simulation model EspaSim 76 ETA 78 EUROMOD 80 TAXBEN 82 Quantitative Partial equilibrium Multi-market models 84 model POLES 86 PRIMES 88 SAFIRE 90 SIMAC 92 Quantitative Sectoral model ASTRA 93 EXPEDITE 96 SCENES 98 TREMOVE 100 Qualitative Other techniques Case study 103 Qualitative Causal chain analysis 10 Qualitative Concept mapping 12 Qualitative/quantitative Concordance analysis 105 Qualitative Delphi method 107 Quantitative Factor analysis 109 Quantitative Gravity analysis 118 Qualitative Impact matrix (Leopold matrix) 14 Quantitative Shift-Share analysis 112 Quantitative Social Accounting Matrix 114 Qualitative Sørensen networking method 116 Judging Qualitative Benchmarking 120 Quantitative Cost-benefit analysis 122 Quantitative Cost-effectiveness analysis 124 Qualitative Expert panels 126 Qualitative/quantitative Multi-criteria analysis 128 Qualitative Risk analysis 130 8

9 2 Fiches database 9

10 Causal chain analysis General description of the technique Causal chain analysis (CCA) looks for the cause of something happening. The technique thus aims to identify the significant cause-effect links between the proposed change in a trade measure and its potential economic, social and environmental impacts. For example, a change in tariff levels will directly alter the pattern of prices facing producers and consumers. Similarly, a rules change in competition policy alters the market conditions for producers and consumers, although the impact on prices will be indirect. Having selected the priority issues to be addressed by the chain, a team of experts examines the immediate causes of the issues and then assesses from which sector of the economy the immediate causes have arisen through a sectoral analysis approach using seven generic sectors. The technique then explores the social and economic reasons for the key resource uses and practices. There is also a simple analysis of existing measures taken to limit the cause. The final step in the technique is a simple analysis of underlying or root causes. The causal chain analysis is very generic and prescriptive in its nature, with an option for public participation. This technique can be used in almost every project, covering various subjects. On the net it is often used in evaluations of plane crashes, more generic transportation evaluations, environmental issues, and in trade related issues. However, it should be used together with quantitative techniques. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring/Analysing data Identifying expected effects/comparing observations Regions all Specific limitations - no universal theory exists - great variation in local circumstances - no spatial impacts or impacts over time are taken into account - complex diagrams are common with this technique Main policy field all 10

11 Specific aspects related to social impact assessment Description the technique can be geared entirely toward the social effects of the policy, because social impacts can be identified as issues for which the causes need to be examined General outputs The output is an overview of causes and linkages through which the effects come into being. Social impacts Employment: yes Job quality: yes Standards and rights related to job quality: yes Social inclusion and protection: yes Discrimination and equality of treatment and opportunities: yes Access to social protection, health and educational services: yes Public health and safety: yes References External link EC Impact Assessment Guidelines 11

12 Concept mapping General description of the technique Concept mapping was first used in firms but is now applied to public policy making and in particular to social programmes. Its purpose is to describe and structure policy effects. A stakeholder group first identifies the actors' representation of the socio-economic reality and all expected impacts from a policy. Then, data processing software is used to obtain a graphic image of these expected impacts, which are clustered according to content and importance and weighted. This image is then again discussed by the stakeholder group. As a final result, concept mapping thus clarifies and grades expected impacts from a public intervention, allowing for the definition of weighted evaluation criteria and eventually impact indicators. In an ex-ante context, it can determine the main objectives and priorities. In an ex-post context, it can draw up the evaluation reference system of criteria and indicators that measure effectiveness and impacts. It makes it possible to conduct collective reflections in large heterogeneous groups and their involvement from the early stages leads the results to be readily accepted. Yet, it is advisable to limit the use of concept mapping to complex programmes with many expected impacts, due to its relatively cumbersome implementation. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring/Analysing data Identifying expected effects/comparing observations Regions all Specific limitations - difficult to constitute the right group of partners with a sound balance - despite the software tool, the risk of dominant opinions prevailing remains real Main policy field all Specific aspects related to social impact assessment Description The group can be asked to focus on social impacts, or these can be part of the overall analysis General outputs graph of clusters of policy effects; possibly weighted evaluation criteria per cluster and impact indicators per evaluation criterion 12

13 Social impacts Employment: yes Job quality: yes Standards and rights related to job quality: yes Social inclusion and protection: yes Discrimination and equality of treatment and opportunities: yes Access to social protection, health and educational services: yes Public health and safety: yes References External link EC Structural Funds - MEANS Collection Complementary techniques the evaluation criteria resulting from concept mapping are weighted, so that they can be used directly as input for multicriteria analysis. 13

14 Impact matrix (Leopold matrix) General description of the technique The impact/leopold matrix is a descriptive technique to map the impacts of a policy and to identify the most important impacts to consider. For this, a matrix is set up with the different components of the policy (rows) and the main expected impacts (columns). Each cell is then given a score according to the importance of the specific impact for the specific policy component. The impact matrix thus ensures that all expected impacts are described, structured and considered before a selection is made for further in-depth examination. Leopold matrices are a specific example of impact matrices. The advantange of the Leopold matrix is that the impacts are quantified. In each box the magnitude of each interaction is indicated by giving a number from -10 (very negative) to +10 (very positive). This number is put in the upper left hand corner. Another number, 1 (minimum) to 10 (maximum) is added in the lower right hand corner, which indicates the real importance of the phenomenon for the given project. Finally all important impacts that have a large numerical for magnitude and real importance will be described and explained. It is best used in combination with other impact assessment methodologies like multi criteria decision making techniques. It can form a clear representation of an impact assessment for policy makers. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring/Analysing data Identifying expected effects/comparing observations Regions all Specific limitations - indirect effects are not accounted for - components are not linked to one another - it is subjective and thus open to bias - spatial impacts or impacts that vary over time are not accounted for - the probability of an impact occurring is not accounted for Main policy field all 14

15 Specific aspects related to social impact assessment Description the technique takes into account the social effects of the policy, however, the underlying mechanisms are not clarified General outputs - a quantified magnitude and real importance of the effects of activities on the proposed conditions - a description and discussion of the important impacts - a clear representation of an impact assessment in a diagram Social impacts Employment: yes Job quality: yes Standards and rights related to job quality: yes Social inclusion and protection: yes Discrimination and equality of treatment and opportunities: yes Access to social protection, health and educational services: yes Public health and safety: yes References External link EC Impact Assessment Guidelines (Impact matrix) 15

16 Logical framework General description of the technique The logical framework structures the components of a policy initiative and provides an in-depth analysis of the internal and external coherence of the policy. It produces a formal matrix presentation of the internal functioning of the policy, of the means for verifying the achievement of the goals and of the endogenous and exogenous factors conditioning its success. It facilitates the formulation of the key stages of a policy and allows for the grading of objectives. Also, assumptions and conditions for success of the policy are clarified. Causal links between resources, results and objectives are identified as well as succes criteria and means for verifying the achievement of objectives. As a result, a monitoring and control system can be set up from the early stage of the policy on. Also, responsibilities of each actor involved in the execution of the policy can be delimited. It is mainly used as a prospective impact assessment tool. It is ideal in the context of simple programmes where the objectives and main actors are clearly identified. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring Identify expected effects Regions all Specific limitations - participation of the actors involved in the policy initiative is needed so that they fill in the framework - difficult to apply to complex projects with multiple aims - risk of designing a programme with vague outlines and mechanisms due to the presence of several decision-makers Main policy field all Specific aspects related to social impact assessment Description all outputs, goals and aims of a policy are mapped into the logical framework, thus the technique can implicitly or explicitly account for social effects General outputs a schedule of all inputs, outputs, goals and aims of a programme, interlinked by means of causal relations 16

17 Social impacts Employment: yes Job quality: yes Standards and rights related to job quality: yes Social inclusion and protection: yes Discrimination and equality of treatment and opportunities: yes Access to social protection, health and educational services: yes Public health and safety: yes References External link EC Structural Funds - MEANS Collection 17

18 Objectives diagram and impact diagram General description of the technique Objective and impact diagrams relate direct outcomes and field impacts with the expected objectives and impacts of the programme. Each objective or impact is presented as logically dependent on a higher row objective or impact and the outputs of the activities appear as a contribution to the overall objective/impact and support the coherence of the objectives and impact system. The construction of this kind of diagrams is meant to observe and analyse and is best used to structure the impact assessment and identify the main evaluation questions. In the objective diagram, the objectives of the policy are identified and classified from global to more detailed operational objectives. In the impact diagram, results, outcomes and impacts of what is intended from the implementation of the objectives system are classified. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring Identify expected effects Regions all Specific limitations - risk of representing the objectives and impacts in an over-simplistic way when trying to provide readable diagrams for presentation Main policy field all Specific aspects related to social impact assessment Description The technique aims at a full list of objectives and impacts, with an emphasis of the most relevant ones in the diagram representation. Social effects are thus considered and could be focused on if desired. General outputs A structured representation of the objectives classified from global to operational and of the impacts, complemented with the (causal) links between them. 18

19 Social impacts Employment: yes Job quality: yes Standards and rights related to job quality: yes Social inclusion and protection: yes Discrimination and equality of treatment and opportunities: yes Access to social protection, health and educational services: yes Public health and safety: yes References External link EuropeAid Evaluation Toolkit 19

20 Problem diagram General description of the technique The analysis of problems is a means to reveal, structure and test the validity of the objectives of a project, a programme or a strategy. The problems that the policy wants to handle are identified, together with the potential causal links between them. Problems are classified going from core problems (according to the higher-order objectives) to smaller issues (according to the lower-order objectives). To draw up a problem diagram, one must also keep in mind that a policy should always be seen within the context in which it is implemented. A problem diagram can be used to organise the evaluation when setting up an objectives diagram directly is not possible. The objectives can then be deduced from the problems. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring Identify expected effects Regions all Specific limitations - a multidisciplinary team of experienced evaluators with knowledge of the theme and the geographical area is needed - quality of the data might be an issue, coming from strategic, political and programming papers when there are no references to data sources - quality of the diagram depends on the identification of the core problem, which is not always clearly stated or identifiable - risk of simplification in tree representation Main policy field all Specific aspects related to social impact assessment Description The technique aims at a full list of problems and objectives, with an emphasis of the most relevant ones in the diagram representation. Social effects are thus considered and could be focused on if desired. 20

21 General outputs A structured representation of the problems classified from core to operational and of the objectives one can link to these problems, complemented with the (causal) links between them. Social impacts Employment: yes Job quality: yes Standards and rights related to job quality: yes Social inclusion and protection: yes Discrimination and equality of treatment and opportunities: yes Access to social protection, health and educational services: yes Public health and safety: yes References External link EuropeAid Evaluation Toolkit 21

22 SWOT analysis General description of the technique A SWOT analysis examines the strengths and weaknesses of the policy initiative as well as the opportunities and threats. By identifying the dominant and determining factors that influence the success of the policy initiative, it thus enables the definition of a more relevant strategy within the context in which the action is to take place. The link between policy initiative and environment is crucial in this technique. Because the technique takes into account the feasibility (S&W) and potential (O&T) of the policy, different possibilities can be classified and changes proposed. It is mainly a descriptive and structuring tool. Use is particularly recommended in the planning of a policy and during its ex ante evaluation, helping to improve the integration of the policy into its context. But also for intermediary and ex post evaluations, it can be used to check the relevance and coherence of the policy. Selection criteria General characteristics Qualitative Sectoral or general Local or global Stage and substage in evaluation or impact assessment Structuring Identify expected effects Regions all Specific limitations - there is a risk that the technique results in a simplistic framework - activities can be classified in a subjective manner, influencing the credibility and usefulness Main policy field all Specific aspects related to social impact assessment Description the technique can be geared entirely toward the social effects of the policy, accounting for the more general context and each stakeholder's position General outputs All types of outputs can be focussed on, but results will depend on the availability of information on these topics (quantitative and qualitative data) 22