Hierarchies of evidence

Transcription

1 Hierarchies of evidence Experiences from Education Policy Debby Lanser

2 Introducing CPB Evidence based policy making: Charted Choices Evidence based policy in practice Hierarchies of evidence What defines a good evaluation process?

3 CPB and a bit about me CPB operates on the intermediary between policy and science CPB, Netherlands bureau for economic policy analysis, is mainly known for its economic forecasts its analyses of the election manifestos of political parties (KiK) its analyses of policy proposals in a number of different ways ex post ex ante scenario s cba s... My experience: Education policy, ex post and ex ante 3

4 KiK: Charted choices Analyses of manifestos of political parties (last election: 10 parties) macro economic outcomes (employment, growth, sustainability) purchasing power special topics health care education infrastructure environment... High impact: policy debate formation of the new cabinet acual policy making 4

5 Pro s and Cons Pro s: Put your money where your mouth is Evidence entering the policy debate Of assistance to political parties and the people how to realise ideals? formation process assessment of parties on objective grounds Urge on more evidence! Cons: Gaming What about new policies?... 5

6 When should/would/could evidence enter the policy process? What is actually social evidence based policy? Goals, mechanisms, relations and causality A long decision process: Politics, formation, departments, legal discourse, parliaments and implementation How to assess evidence? To consider: literature reviews ((inter)national evidence and mechanisms think big, start small (experiments) limited interest after a policy is implemented (ex ante, ex post) external and internal validity 6

7 An example: teacher performance pay International literature: theoretical empirical: increasing student performance Charted choices: right climate, nice effects Made it through the formation process Difficulties: External validity no clear criteria, no evaluations structure, e.g. lacking performance interviews much opposition 7

8 Hierarchies of evidence ( ex post ) Different methods, different quality Social and lab experiments: treatment and control group Observables and non observables Natural experiment Regression discontinuity Difference in difference Monitoring, surveys, case studies. 8

9 What defines a good evaluation practice: Pro s: isolating the causal effect Con s: policymakers: not ethical to draw people in experimental and control groups why withhold people a good measure implementation: hampered by good intentions, unintentionally changing research designs circumstantial evidence outcome is an effect, no description of underlying mechanisms or critical success factors 9

10 Examples: experiments on school drop outs Experiments on school drop outs on forehand intermediate vocational schools signed up for the evaluation underscribing the method of drawing students and a prescribed minimum number of participants. a selection committee evaluated the entries five intermediate vocational schools were picked First experiences: Good intentions well organised Difficulties: collecting the right data, implemenation within school 10