Flanders Environment Report M IR A-PE 2OO5 POLICY EVALUATION

Transcription

1 53 Flanders Environment Report M IR A-PE 2OO5 POLICY EVALUATION i

2 Flanders E n v iro n m e n t R eport: policy evaluation

3 C h a irm a n : Rudi Verheyen (IM) P r o je c tm a n a g e r : Marleen Van Steertegem S e c re ta ry : Philippe D Hondt (VMM) M e m b e rs f o r th e F le m is h C o u n c il f o r S cie n ce P o lic y : Patrick Meire (IM) Chris VinckierfK.LL Leuven) M e m b e rs o f th e B o a r d o f S ecretaries-general: Veerle Beyst (APS) Ludo Vanongeval (AMINAL) M e m b e rs o f th e F le m is h N a tu re a n d E n v iro n m e n t C o u n c il: Luc Goeteyn (MiNa Council) ) a n Tu r f (Bond Beter Leefmilieu vzw) T e a m m e m b e rs : M yriam Bossuyt lohan Brouwers C aroline De Geest Hanne Degans Sofie Janssens S tijn O verloop Bob Peeters Lisbeth Stalpaert Barbara Tielem an Hugo Van Hooste Jeroen Van Laer Els van W alsum Erika Vander Putten M e m b e rs f o r th e F le m is h S o c io -E c o n o m ic C o u n c il: Annem ie Bollen (SERV) Peter Van Humbeeck (SERV) In d e p e n d e n t e x p e rts : RikAmpe(Vito) Jeroen Cockx (GMO) Geert De Blust (IN) M yriam Dumortier(NARA, IN) Rudy Herman (AWI)

4 Flanders Environment Report: policy evaluation MIRA-PE 2OO5 Presented on 21 O ctober 2005 to Kris Peeters, F le m is h m in is te r f o r P u b lic W o rk s, E n e rg y, th e E n v ir o n m e n t a n d N a tu re M arleen Van Steertegem, fin a l editing ' A VLI2 (vzw) VLAAMS INSTITUUT VOOR DE ZEI FLANDERS MARINE INSTITUTE Oostende - Belgium VLAAMSE MILIEUMAATSCHAPPIJ

5 Marleen Van Steertegem (fin a l editing) FLANDERS ENVIRONMENT REPORT: POLICY EVALUATION M IR A - P E 2O O 5 72 pp., 24.5 cm D/2006/6871/ , Vlaamse Milieumaatschappij, Aalst (Belgium) The use o f texts fro m the report w ith acknowledgem ent to the source is encouraged. Layout and cover design SignBox, Peter De Roy 81 Betty Reyniers Typesetting Ourtype Versa Linguistic correction Stein Pée Translation Oneliner Language 81 ebusiness Solutions b v b a (Belgium) Printed by Sintjoris nv, Merendree (Belgium) Contact Flemish Environment Agency - m ir a (v m m ) Van Beneden laan 3 4, B Mechelen, Belgium tel: fax: mira(a)vmm.be Printed on 50 % post-consumer recycled paper, chlorine-free and without optic bleach. The illustratio ns on the cover and d iv id in g pages are adaptations o f pages from the Linnean Herbarium a t the Swedish Museum fo r N atural H istory in Stockholm.

6 FOREWORD W hat can w e learn fro m p o lic y e va lu a tio n s? is th e title o f a new p u b lic a tio n o f th e Swedish E n viro n m e n ta l P ro te ctio n Agency, in w h ich a round tw e n ty e n v iro n m e n ta l p o lic y eva lu a tio n s are re p o rte d and discussed1. The title p e rfe ctly sum m arises th e pu rpose o f p o lic y e va lu ations: to learn fro m th e m ; to discover m echanism s th a t w o rk o r id e n tify p ro b le m s in o r d e rto solve th e m. Before yo u lies th e late st p o lic y eva lu a tio n re p o rt o f th e Flem ish E n viro n m e n t Agency, in s h o rt MIRA-PE The re p o rt o ffe rs an ove rview o f 24 eva lu a tio n studies o f th e Flem ish e n v iro n m e n ta l p o lic y carried o u t d u rin g th e past tw o years. It includes: in s tru m e n ts, p o lic y m easures, ta rg e t groups, effects, side effects, ju s tic e, know le dge, pow er, u n w illin g n e ss, a m b itio n... In sum m ary, th e re p o rt is a b o u t (e n v iro n m e n ta l) p o lic y and h o w th is p o licy w o rks (o r does n o t w o rk). i Swedish Environmental Protection Agency (2004) 21 evaluationsw hat do they teach us? Naturvdrdsverket, Stockholm ( M IRA-PE This is th e second MIRA-PE re port. It is its e lf part o f a learning process in th e shape o f a step-by-step plan. MIRA-PE 2003 contained an in tro d u c tio n to po licy concepts and fo u r in -d e p th evaluations o f aspects related to th e w ate r and waste policy. They w ere conceived asa basis fo r a m o re extensive research progra m m e regarding po licy evaluation o n th e o n e hand and fo r th e cre ation o f an expertise n e tw o rk in Flanders on the oth er. MIRA-PE 2005 bu ild s on th is: it offers five in -d e p th evaluations w hich refer to five co m p o n e n ts o f the po licy chain. B ut th is latest re p o rt goes even further. An inve n to ry o f th e key e valuation studies on th e Flem ish e n viro n m e n ta l po licy provides m o re in sig h t in to th e flaw s o fth e system. Moreover, less w ell-know n environm ental policy areas are given attention as w ell. In a d d itio n, th e e xp ertise n e tw o rk is fu rth e r expanded. The MIRA-PE process and (th e p u b lic a tio n s o f) th e C entre o f Expertise on E nviro n m e n ta l Policy Sciences, a m o n g o th e rs, have c o n trib u te d to a g re a te r fa m ilia rity w ith th e a ctivitie s o f researchers fro m d iffe re n t research centres. M oreover, these researchers w ere pro vid e d w ith a fo ru m in w h ic h to exchange ideas and coo pera te. Som e experts c o m m itte d the m selves as a u th o rs and researchers o f a MIRA chapter, o th e rs chose to p ro vid e th e MIRA-PE a u th o rs w ith th e essential (q u a n tita tiv e and q u a lita tiv e ) data. In fact, h ig h -q u a lity and o b je c tiv e po licy e va lu ations re q u ire th e c o o p e ra tio n o f people involve d in th e po licy. T h e ir e n thusiasm and efforts are extrem ely valuable, instructive and an exam ple. We w o u ld like to th a n k everyone: th e MIRA team fo r its c o o rd in a tio n o f th is re p o rt, th e a u th o rs fo r th e ir research and th e ir c o m p e llin g synthesis, th e revisors fo r th e ir o p in io n s and suggestions and th e MIRA S teering G roup fo r its c ritical re fle c tio n and cre ativity. O nce m o re, a h ig h -q u a lity re p o rt has been pro d u ce d, tha nks to th e e ffo rts o f many.

7 It is said th a t e xp erience is th e best teacher. This is c e rta in ly tru e fo r re p o rtin g on e n v iro n m e n ta l p o lic y e va lu a tio n. In o th e r co u n trie s, m o re and m o re re p o rts on p o licy e va lu a tio n are p u b lish e d. H ow ever, even th e re th e tra d itio n is lim ite d and best practices' are scarce. MIRA b e lo ngs to th e (select) clu b o f leaders in th is respect. U n fo rtu n a te ly, leaders o fte n have to search fo r th e rig h t way: n o ta n easy task. We are happy to take on the challenge. Frank Van Sevencoten A d m in is tra to r-g e n e ra l o f th e F lem ish E n v iro n m e n t A gen cy (VMM) RudiVerheyen C h a irm a n o f th e M IR A Steering Croup September 2005

8 CONTENTS M IRA-PE 2005 Sum m ary g About MIRA-PE il 1 W hat is MIRA-PE? ti 2 W hat can y o u fin d in th is re p o rt? 12 3 W h a t is th e fu n c tio n o f MIRA-PE? 12 4 W here d o w e w a n t to go? 14 5 W hat are typica l e le m e n ts o f MIRA-PE? 15 6 W ho is involved in MIRA-PE? 17 7 W ho is MIRA-PE aim e d at? 18 References 18 Environmental policy under THE MICROSCOPE: 24 EVALUATIONS 21 In tro d u c tio n 21 1 In ve n to ry 22 2 D iscussion 26 3 Evaluation sum m aries 38 Annex: In ve n to ry reference lis t References 65 Revisors 65 Explanatory lis t o f a b b re via tio n s 6 9 A b o u t th e Flem ish E n viro n m e n t Agency 70 File on Flanders 71

9

10 Summary MIRA-PE is th e bian nua l p o lic y e va lu a tio n re p o rt o f th e Flem ish E n viro n m e n t Agency (PE standing fo r 'P olicy Evaluation ). Policy evaluation is d efined as follow s : the scientific analysis o f a sp e cific p o lic y th e m e w h ich is evaluated o n th e basis o f crite ria and serves as a s ta rtin g p o in t fo r th e fo rm u la tio n o f re com m e n d a tio n s. M I RA-PE 2005 MIRA-PE 2005 o ffe rs -a fte r a c h a p te r w ith m o re in fo rm a tio n on th e characteristics o f th is re p o rt-a n o ve rvie w o f th e eva lu a tio n studies o f(su b )a re a s o f th e Flem ish e n v iro n m e n t p o lic y p u b lish e d betw een 1 July 2003 and 31 May W ith th is inven tory, MIRA-PE 2005 fu lfils th e o b je ctive o f b e in g a w o rk o f reference w ith regard to po licy eva lu a tin g studies in Flanders. A fter a discussion o f the inventory, there isa sum m ary o f the 24 policy evalua tio n s stru ctu re d in files. The sum m ary system atically shows th e m a tte r th a t was researched, th e m e th o d, th e fin d in g s and th e re co m m e n d a tio n s o fth e p o lic y eva lu ation. The discussion o ft h e in v e n to ry includes, in th is o rd e r: th e typ e o f policy, th e typ e o f in s tru m e n t, th e typ e o f e va lu a tio n, th e e n v iro n m e n ta l th e m e s (lin ke d to th e Flem ish R egion s e n v iro n m e n ta l s p e n d in g ), th e eva lu a tio n crite ria used, th e o rd e rin g party and th e e va lu a tin g in s titu tio n s. O ne o fth e fin d in g s is th a t all p o licy and in s tru m e n t types are researched m o re o r less equally. O n ly th e c o o rd in a tin g (e n v iro n m e n ta l) p o licy is researched less. Secondly, a tte n tio n is paid especially to ex-post and ex-nunc evaluatio n s and process and p ro d u c t e va lu ations. In general, e n v iro n m e n ta l th e m e s fo r w h ich a lo w e r b u d g e t is ava ila ble are evaluated less. One o fth e e xce ptions is th e w a te r policy, w h ic h absorbs m o re th a n h a lf o f th e e n v iro n m e n ta l b u dget, b u t is evaluated re la tive ly little. F u rth e rm o re, o n ly a lim ite d n u m b e r o f e va lu ations ta lk a b o u t effectiveness. The g re a t m a jo rity o ft h e e va lu a tio n s re fe rto th e m u tu a l c o o rd in a tio n, in te g ra tio n, k n o w l edge and capacity a m o n g th e p e o p le involve d in th e p o licy etc. M o st p o lic y e va lu ation s w ere ord e re d by th e g o v e rn m e n t. M o st o fth e researchers w h o carried o u t th e evaluatio n s w o rk fo r a u n ive rsity. The discussion also investigates w h e th e r th e re is any c orrespo nde nce betw een th e in v e n to rie d p u b lic a tio n s w ith regard to m e th o d o lo g y o r c o n te n t, w h e th e r th e y are ne u tra l and in d e p e n d e n t, w h e th e r th e y are o f high q u a lity and w h ic h im p a c t th e y could have on th e fo rm u la tio n o f a policy. From th e in ve n to ry, it becom es clear th a t th e re are a n u m b e r o f h o t to p ics: several studies researched e n fo rc e m e n t, th e in te rn a tio n a l e n vironm ental policy and the role o fth e Flemish Region in it and the inte gratio n o fth e various g o v e rn m e n t bodies. The e ffe c t o fth e s tudies on th e p o licy is varia ble: w e suspect th a t th e eva lu a tio n s in itia te d by th e g o v e rn m e n t its e lf could have q u ite a b ig im pa ct, p ro v id in g th e y are carried o u t by th e people in charge o fth e policy.

11

12 About MIRA-PE.. What is MIRA-PE? MIRA-PE Is th e Flem ish E n viro n m e n t Agency s bian nua l re p o rt on th e eva lu a tio n o fth e Flem ish e n v iro n m e n t p o lic y (PE stands fo r Policy E valuatio n ). Policy e va lu a tio n is d e fin e d as fo llo w s in th is re p o rt: th e s c ie n tific analysis o f a sp e cific p o lic y th e m e w h ic h is evaluated on th e basis o f crite ria and serves as a s ta rtin g p o in t fo r th e fo rm u la tio n o f re co m m e n d a tio n s. The c o n te n t o f MIRA-PE is d e term in ed by po licy them es. A policy them e is (part o f a) policy area (e.g. waste policy) and can refer to one o r m ore instrum ents a n d /o r one o r m o re environm enta l issues fo r w hich a policy is carried ou t. M oreover, it can be a b out the agenda-setting, objectives, p la n n in g, e x e c u tio n, etc. o f th a t po licy. O th e r areas in v e s tig a te d by MIRA-PE are th e consequences o f t h e p o lic y o n th e b e h a v io u r o ft h e ta rg e t g ro u p s (o u tc o m e ) a n d /o r o n th e e n v iro n m e n t (e n v iro n m e n ta l e ffe c t). The c o m in g in to b e in g o f MIRA-PE is q u ite a lo n g story. Even th o u g h alre ady in 1995 th e re g io n a l le g is la to r o rd e re d th e e n v iro n m e n ta l re p o rtin g d e p a rtm e n t to m ake p o lic y e va lu ations, it was n o t u n til 2000 th a t th is task was really started. Therefore, the first e n vironm ental reports did contain critical analyses o f the e n vironm e n t policy, b u t th e y lacked in -d e p th e va lu ations. In May 2000, th e th e n m in is te r o fth e E n viro n m e n t asked the chairm an o fth e MIRA steering group to re port syste m a tic a lly on in -d e p th p o lic y e va lu ations. The steering g ro u p o p te d fo r a step-by-step plan, s ta rtin g by d ra w in g up an in v e n to ry o f e x is tin g p o lic y e v a lu a tio n s and m e th o d s.t h e ste p -b y-ste p plan had to lead to th e c re a tio n o f an e xp ertise n e tw o rk (see paragraph 4 W here d o w e w a n t to go?). MIRA MIRA stands for Flanders Environment Report ('Milieurapport Vlaanderen' in Dutch). MIRA is coordinated by the Flemish Environment Agency (VMM) which has hired a project team for this purpose. MIRA s mission has been determined by decree and is threefold: a description, analysis and evaluation ofthe existing state of the environment; an evaluation ofthe environmental policy conducted up to now; a description o f the developments expected in the environment in case o f unchanged as well as amended policy in terms of a number of scenarios that are thought to be relevant. The study ofthe state o f the environment is presented in the annual MIRA-T reports. In 2000, the first full scenario report (MIRA-S 2000) was published. The first policy evaluation report (MIRA-PE) was published in une All publications, figures and indicators from MIRA can be consulted on the website (available in English).

13 The results o f this inventory and o f some in-depth policy evaluations was the subject of th e firs t BE re p o rt in T h e g o v e r n m e n t a n d I T S I N S T R U M E N T S A steering policy instrument denotes an instrument used by the government to persuade, force or encourage a target group or another authority to implement a measure or to enable itself to take measures. Examples: Royal Decrees, decrees, covenants, licences, h ie s, subsidies, awareness campaigns, p revention... A direct policy instrument denotes an effective activity o f the is aimed at the creation of a specific situation. Examples: the construction o f sewers, the collection o f waste, the construction o f roads, the construction o f container parks, the planting o f a row o f trees.., Supporting instruments help with the choice o f steering and direct policy instruments. Examples: plann ing, market creation, research and development, monitoring, information management... government which intervenes in the physical environment and 2. What can you find in this report? This English versio n o f MIRA-PE conta in s an in v e n to ry and analysis o fth e eva lu a tio n studies o fth e Flem ish e n v iro n m e n t p o lic y p u b lished betw een 1 July 2003 and 30 May The D utch versio n conta in s an a d d itio n a l part c o n sistin g o f five in -d e p th po licy eva lu ations, selected fro m th e in v e n to ry and draw n up a cco rd in g to a fixed stru ctu re. These chapters have been selected in such a way th a t each on e evaluates a d iffe re n t elem ent in the policy chain. 3. What is the function ofmira-pe? W ith MIRA-PE 2005, w e w a n t to cover th re e d iffe re n t o b je ctives: re p o rtin g on e n v iro n m e n t p o lic y studies, fo llo w in g up th e e n v iro n m e n t p o licy p la n n in g and b u ild in g up expertise. A WORK OF REFERENCE MIRA-PE 2005 w ants to c o n s o lid a te its fu n c tio n o f a w o rk o f reference. M o re th a n in MIRA-PE 2003, w e o ffe r an ove rview, a state o f affairs o fth e e n v iro n m e n ta l p o licy evaluations in the Flemish Region.

14 POLICY PLANNING MIRA-PE aim s at o ffe rin g added value fo r th e fo llo w -u p o fth e e n v iro n m e n ta l Policy Plan by c o m m e n tin g on th e m easures in clu d e d in th e plan. T h ere fore, th e re p o rt m u st fit th e p o lic y m akers know le d g e re q u ire m e n ts as closely as possible, i.e. it m u st be as p o lic y re le vant as possible. A m o n g o th e r th in g s, w e have a d ju sted th e d iv is io n o fth e po licy e va lu a tio n s as m u ch as po ssib le to th e latest E nviro n m e n ta l Policy Plan (M IN A-Plan 3). MIRA-PE w a n ts to h e lp s u p p o rt th e m a n a g e m e n t cycle as presented in th e M IN A -p la n, concre te ly by re p o rtin g data fo r th e eva lu a tio n o fth e p o lic y (fig u re 1). Figure 1: M a n a g e m e n t cycle in M IN A -P lan 3 and th e place o f MIRA-PE in it > Policy preparation Policy evaluation Policy implementation MIRA-PE Source: Environmental Policy Plan , p. 322 LEARNING PROCESS MIRA-PE 2005 aim s a t c o n trib u tin g to th e le a rn in g process related to p o licy e va lu a tio n. This le a rn in g process, w h ic h is an u p c o m in g success in th e Flem ish p o lic y practice and especially in th e e n v iro n m e n t a d m in is tra tio n, provides re fle c tio n on th e p o licy and raises a n u m b e r o f qu e stio n s. Are w e on th e rig h t track? W hat m e th o d s are useful? Q u e stions can also be asked re g a rd in g th e usefulness o fth e p o lic y e va lu ations. Can th e p o licy e va lu ations presented here o n ly be a p p lie d to th e ir s pecific o b je ct? O r can co n clu sio n s and m e th o d s also be used in o th e r p o licy areas? N o t su rp risin g ly, each p o lic y th e m e has a n u m b e r o f specific characteristics w h ic h m ake g e n e ra lisatio n s to o th e r p o lic y areas d iffic u lt. For instance, each p o licy area has its o w n histo ry, actors, sp e cific in s titu tio n s, le g is la tio n, p o litic a l se n s ib ilitie s, etc. Flowever, th e m e th o d s, eva lu a tio n crite ria and assu m p tio n s in th e present re p o rt can be a p p lie d to o th e r (e n v iro n m e n ta l) p o lic y areas. Lessons w e can draw fro m p o licy e va lu a tio n s in a p a rtic u la r p o lic y area may also be valid fo r o th e r p o licy areas. T h ere fore, th e in fo rm a tio n in th is re p o rt can be used to answ er th e fo llo w in g un ive rsal p o lic y qu e stio n s: Fiow can w e d e te rm in e th e e ffe c t o f a p ro g ra m m e o n th e ta rg e t group? W hat d iffic u ltie s can acco m pan y th e d e v e lo p m e n t o f a re g io n -o rie n te d in te g ra te d policy and how can we manage them?

15 H ow can w e p re d ic t w h e th e r a re o rg a n isa tio n o f services m ig h t be useful? H ow can th e observance o f laws be guaranteed once th e y have e n tere d in to force? H ow can E uropean (in te rn a tio n a l) le g is la tio n be transposed successfully in to th e Flem ish con te xt? W h a t are th e c o n d itio n s fo r success? H ow can th e im p le m e n ta tio n o f a p ro g ra m m e be o p tim iz e d? W h a t are th e typ ica l problem s? W h a t are success factors in th e in te g ra tio n o f p o licy levels (e.g. fro m E uropean to Flem ish, regional to local level)? H ow can European d ire ctive s be transposed c o rre ctly fro m a legal p o in t o f view? 4. Where do we want to go? For th e re a lisatio n o f its s ta tu to ry p o lic y eva lu a tio n m ission, MIRA w o rks w ith a step -bystep plan. A firs t step consists o f d ra w in g up an in v e n to ry o f e x is tin g research in Flanders and c o n tin u in g research o n ce rta in p o lic y them es. These eva lu ation s p ro vid e th e basis fo r th e ne xt step, in w h ic h m o re extensive and m o re general analyses and assessm ents o f e n v iro n m e n ta l p o lic y areas are m ade. This is possible by m o b ilis in g and s tru c tu rin g an exp e rtise n e tw o rk fo r p o lic y e va lu ations. This process is c u rre n tly in progress and MIRA-PE 2005 w ants to c o n trib u te to it. By m eans o f a m a trix w e o ffe r you a visual ove rview o fw h ic h p o lic y th e m e s have been tre ated in MIRA-PE 2003 and MIRA-PE 2005 a nd, th e re fo re, w h ic h th e m e s s till need to be covered (ta b le i).t h e n u m b e r o f'e m p ty ' cells speaks fo r itself. We re fe ry o u to th e in v e n to ry (n e xt c ha pter) o ft h e present re p o rt fo r fu rth e r c o m m e n ts o n th is m a trix and the corresponding concepts. Table 1: M a trix o f evaluated p o lic y to p ic s in MIRA-PE 2003 and MIRA-PE 2005 policy type* environmental topic general climate waste manure nature water region international integration target group coordination photochemistry ozone layer acidification hazardous substances soil nuisance fragmentation region = region-oriented policy; in tern atio n al = international policy; integration = integrated govern m en t policy; targ et group = target group policy; coordination = coordinating policy

16 T y p e s o f p o l i c y e v a l u a t i o n P o l i c y e v a l u a t i o n a c c o r d i n g P o l i c y e v a l u a t i o n a c c o r d i n c t o t i m e TO THE SUBIECT plan evaluation: review o f plans and/or goals before or after their implementation process evaluation: review o f policy processes, what concerns the development o f the policy as well as its implementation product evaluation: review of the policy results ex-ante evaluation: evaluation before implementation o f the policy ex-nunc evaluation: evaluation during implementation o f the policy (also called ex-interim) ex-post evaluation: evaluation after implementation o f the policy 5. What are typical elements of MIRA-PE? MIRA-PE 2005 ABOUT MIRA-PE The re p o rts o n th e State o ft h e E n viro n m e n t in th e Flem ish Region (MIRA-T) are draw n up on th e basis o f in d ica to rs and have a fixed fra m e w o rk fo r analysis: th e d is ru p tio n chain o r DPSIR chain. MIRA-PE also conta in s such re cu rrin g e lem ents, co n cre te ly th e po licy chain as a fra m e w o rk fo r analysis, th e eva lu a tio n crite ria and th e fixed stru ctu re. First, w e w ill ta lk a b o u t th e p o lic y chain. THE POLICY CHAIN AS A FRAMEWORK FOR ANALYSIS The po licy chain indica tes th e p o lic y process (fro m th e s ta rt o fth e p o licy up to its results). It a llo w s us to unravel th e c o m p le x ity o f th e p o licy and serves asa g u id e fo r a good understanding o fth e present report. Figure 2 visualises the policy chain. Figure 2: The p o lic y chain as a fra m e w o rk fo r analysis fo r MIRA-PE social and political issues socioeconom ic factors European legislation p o lic y process e n v ir o n m e n t d is r u p tio n c h a in plan w ith objectives program m e and budget output enforcem ent outcom e behaviour o f target groups environ m ental effect pressure-s itu a t io n - im pact operationalisation im p lem en ta tion environm ental processes Source: Wilkinson (1999).

17 T h e p h a s e s i n t h e p o l i c y c h a i n ( f i g u r e 2 ) The chain starts with one or more social and political issues which are put on the agenda o f the policy makers. This starts a process, the policy process, which consists of three phases. Each phase leads to a concrete result (displayed in a circle). The first phase is the planning phase. Solutions for the social and political issues are designed, compared and discussed. This should normally result ina plan with clear objectives. In the second phase, the operationalisation phase, the plans become more concrete: a programme is drawn up in keeping with the budget in order to ensure optimal implementation with the available human and financial resources. After that, the plan is implemented (phase 3). In the field, we need to see a concrete output o f this: e.g. a planted row of trees, a granted licence, a builtwind park, an imposed fine... This output is related to the environmental disruption chain or DPSIR chain which is also displayed in figure 2. People adjust their behaviour, which contributes indirectly to a decrease in environmental pressure, leading in turn to an improved environmental situation and possibly havinga positive impact on people, nature and the economy. The output can also influence the environment directly, without the need for a change in behaviour. However, apart from the policy chain external factors play a role as well. These are harder to control with a policy but their influence on environmental disruption is just as big - or even (a lot) bigger. These factors must not be forgotten because they demonstrate the government's fallibility when it comes to controlling social processes and therefore the environment. By m eans o f arrow s, fig u re 2 shows th e causal co n n e ctio n s betw een th e c o m p o n e n ts o fth e p o licy and d is ru p tio n chains w ith o u t d e scrib in g th e m e x p lic itly. This approach can o n ly be used w h e n th e re is a p o lic y th e o ry w h ic h shows causal re la tio n s (e.g. p o licy in s tru m e n t w ith o u tp u t x leads to im p ro v e m e n t in th e w a te r q u a lity y, w h ic h results in a recovery o f the fish po p u la tio n by z %). However, these p o lic y th e o rie s are rarely ava ila ble because th e Results of the policy u n d e rly in g p o lic y processes (fro m th e p o lic y chain) and th e social an d e n v iro n m e n ta l processes (fro m th e output: government contributions (e.g. amount o f subsidies, e n v iro n m e n ta l d is ru p tio n c h a in ) are in s u ffic ie n tly granting a licence) kno w n o r because th e re are to o m any u n c o n tro lla b le outcome: change in behaviour in the target groups (e.g. external factors. local authorities, citizens, companies) environmental effect: improvement / deterioration o f I f a p o lic y eva lu a tio n w ants to e xp la in th e appearance the environment (and in a later stage nature) as a result of o r non-appearance o f be haviou ral, e n v iro n m e n ta l o r the policy (sometimes also called environmental impact) e co lo gical effects, it n o rm a lly needs to o p en and analyse th e black boxes' o f those be haviou ral, e n v iro n m e n ta l and e co lo gical processes. T h e re fo re, in th e area o f e n v iro n m e n ta l and nature policy, p o lic y eva lu a tio n is nearly always a kind o f process evaluation. EVALUATION CRITERIA A n o th e r typ ica l aspect o f p o lic y e va lu a tio n s and MIRA-PE are c r ite r ia. It is clear th a t a m ere analysis o f a p o lic y th e m e is n o t an e va lu a tio n. An eva lu a tio n requires a ju d g e m e n t and th is is m ade u sin g c rite ria. The m o st o b vio u s c rite rio n is effectiveness. This s h o u ld p ro vid e an answ er to th e q u e stio n w h e th e r th e in s tru m e n t o r m ix o f in s tru m e n ts has c o n trib u te d to an in te n d e d change in b e h a vio u r o r to an e n v iro n m e n ta l im p ro v e m e n t. O th e r c rite ria are e ffic ie n c y (was th e o u tp u t o b ta in e d w ith a m in im u m cost?) and ju s tic e

18 (does th e in s tru m e n t ensure a fa ir d is trib u tio n o f pleasure and pain?). The crite ria especially focus on the products (results) o fth e policy. As described, explanations fo r the results m ust be sought in the preceding processes. Those processes have to be evaluated as w e ll, so th a t an e xp la n a tio n can be fo u n d fo r success and fa ilu re. E valuation crite ria in clu d e : c a p a c ity (d o th e actors have th e co rre ct and s u ffic ie n t resources to carry o u t th e ir task?), k n o w le d g e Ev a l u a t io n c r iter ia (d o th e actors kn o w w h a t is expected? has th e re been... enough research in to w hich target groups a n d /o r alter- n a tive s are necessary?), a c c e p ta t io n (is th e o b je c tiv e accepted? are the organisation and the m ix o f instrum e n ts accepted?), in te g r a tio n (are th e o b je ctive s tu n e d in to each o th e r? are th e re o p p o s ite forces w ith in and o u t- side the a d m inistra tion?). Criteria by which policy results are judged: effectiveness, efficiency, justice Criteria by which policy processes are judged (and its results explained): capacity, knowledge, acceptance, transparency, integration, mutual coordination... The R egulatory M a n a g e m e n t U n it o fth e Flem ish m in is try lists e ig h t q u a lity c rite ria fo r fu tu re legislation (Guide Characteristics o f good legislation, 2003): necessary and effective; approp riate and balanced; feasible and enforceable ; law ful; co h e re n t; sim ple, clear and accessible; w e ll-fo u n d e d and n e g o tia te d ; MIRA-PE 2005 ABOUT MIRA-PE perm anently relevant and topical. The re p o rt w ill re fe r to th e c rite ria and h o w th e y are used at re gular tim e s. FIXED STRUCTURE MIRA-PE n o t o n ly re p o rts on e x is tin g p o licy eva lu ations, b u t th e MIRA tea m so m e tim e s also in itia te s in -d e p th p o lic y e va lu ations. These p o lic y e va lu ations have to be prepared according to a fixed structure (table 2). T able 2: S tru ctu re o f an in -d e p th p o lic y e va lu a tio n in itia te d by MIRA Introduction 3 Analysis 1 Situation 4 Evaluation 2 Concepts & methods 5 Conclusions & recommendations 6. Who is involved in MIRA-PE? MIRA-PE 2005 is th e re sult o f a co o p e ra tio n betw een a u thors, revisors (lecturers) and th e MIRA team. 116 experts fro m universities, research institu tes, interest groups, ad m inistra tio n s, etc. have p u t th e ir expertise a t th e disposal o fth e present re p o rt.

19 The a u th o rs w rite th e chapters. W hen p o lis h in g th e ir texts, th e y take in to a cco u n t the remarks and suggestions o fth e readers and the project team. A t agreed tim es, the revisors undertake a critical revision o fth e authors w ork. They are m e n tio n e d a t th e end by nam e and a ffilia tio n. The MIRA tea m o ft h e Flem ish E n viro n m e n t Agency is re sponsib le fo r th e fo llo w -u p w ith regard to th e c o n te n t, th e c o o rd in a tio n and th e e d itin g o f e n v iro n m e n ta l re p o rts and elabora tes c o n te n ts to g e th e r w ith th e authors. A fte r e d itio n by th e p ro je c tte a m, th e fin a l te x t is s u b m itte d to th e a u th o rs and th e revisors fo r a pproval. The s ta tu to rily d escribed MIRA s te e rin g g ro u p is re sponsib le fo r ste e rin g th e c o n te n ts o fth e re p o rtin g process. 7. Who is MIRA-PE aimed at? The re p o rt is a im e d a t p o lic y m akers (m e m b e rs o f p a rlia m e n t, ca b in e t m e m bers, those re sponsib le fo r p la n n in g ), p o lic y executors and o th e r people involve d in th e po licy (e.g. ta rg e t g ro u p representatives). They are th e ones w h o have to tackle th e p ro b le m s id e n tifie d by th e re p o rt, as w e ll as s tim u la te th e success factors. The re p o rt s re c o m m e n d a tio n s can h e lp th e m w ith th e ir task. M oreover, o th e r persons w ith an in te re s t in th e p o lic y (o r a c ritica l v ie w on it) m ay fin d th e re p o rt useful as w e ll. A fte r a ll, a p o lic y has consequences fo r every com p a n y and every citizen. An in tro d u c tio n to th e concepts, m e th o d s and approach o f p o licy e va lu a tio n s can be fo u n d in cha p te r i o f MIRA-PE 2003 (d o w n lo a d a b le a t w w w.m ilie u ra p p o rt.b e ). R e f e r e n c e s Gids Kenmerken van een goede regelgeving (2003) (Guide characteristics of good legislation). Kenniscel Wetsmatiging (Regulatory Management Unit), Ministerie van de Vlaamse Gemeenschap, Brussel. MINA-plan 3 (2003) Milieubeleidsplan (Environmental Policy Plan). Ministerie van de Vlaamse Gemeenschap, Brussel. Wilkinson D. (1999) The contribution o f EU monitoring and reporting on 'sound and effective environmental policymaking. First draft paper for the EEA-project Towards a new EU framework for reporting on environmental policies and measures (project manager M. T. Ribeiro), z i September 1999, eu.int/rem/en/ soundeffective.pdf.

20

21

22 Flemish environmental policy under the microscope: 24 evaluations Barbara Tieleman, M IR A, VM M INTRODUCTION The in v e n to ry o f e n v iro n m e n ta l p o licy e va lu ations ( ) as a p a rt o f th e firs t P olicy E valuation R eport (M l RA-PE 2003) m e t w ith general approval. We w ill c o n tin u e on th is é la n in MIRA-PE 2005 by p ro v id in g an ove rview o f th e p u b lished p o licy e va lu a tio n s fo r the period The p u rpose o f th is in v e n to ry is to re in fo rce its fu n c tio n asa re fe re n c e w o r k o n e n v ir o n m e n ta l p o lic y e v a lu a tio n s. In th is d o c u m e n t w e th e re fo re p re s e n ta n ove rview o f th e po licy evaluation studies, published betw een 1 July 2003 and 30 May 2005 (part 1, Inventory). General conclusions on the fu ll overview are draw n based on particular characteristics. These co n c lu s io n s are presented in part 2 'D iscussion'. In p a rt 3 w e p ro vid e a sum m ary o f each p u b lic a tio n. As regards th e in v e n to ry and th e discussion, w e w ere in sp ire d by a s im ila r in v e n to ry o f th e Swedish E nviro n m e n ta l P ro te ctio n Agency, The exact reference o f each in v e n to ry p u b lic a tio n can be fo u n d in th e annex to th is chapter.

23 i. Inventory t.i. SELECTION OF THE PUBLICATIONS In p re p a ra tio n o f th e in ve n to ry, w e e xp lo re d th e Flem ish U n ive rsitie s' databases (in c lu d in g s u p p o rt centres), th e e n v iro n m e n ta l a d m in is tra tio n, th e p o lic y advisory agencies, th e advisory bo dies SERV and M ina C oun cil, th e Flem ish U n io n fo r C ities and M u n ic ip a litie s (VVSG) and th e e n v iro n m e n ta l lib ra rie s n e tw o rk (Feinet). We c o n sulte d a large v o lu m e o f in fo rm a tio n resources discussing e n v iro n m e n ta l policy, such as a rticle s, re ports, books and o th e rs. The pu rpose o f th e in v e n to ry was to p ro vid e a co m p le te lis t o f h ig h -q u a lity p o licy eva lu ations. M o st p u b lic a tio n s are easily categorised as p o lic y eva lu a tio n s, b u t th is d o e sn t h o ld fo r all o f th e m. W e th e re fo re used th e fo llo w in g g u id e lin e s as s e le ctio n crite ria : A p o lic y eva lu a tio n s h o u ld co m p ly w ith th e characteristics o f th e d e fin itio n o f a p o licy e va lu a tio n, i.e. be a scientific analysis (no personal op in ions, no descriptions); have e n v iro n m e n ta l p o lic y as a central th e m e (n o in vestig a tio n s w ith regard to an e n v iro n m e n ta l p ro b le m w ith o u t c o n sid e ra tio n o f p o licy aspects; no in ve stig a tio n s w ith regard to o th e r p o licie s th a n th e e n v iro n m e n ta l p o lic y - in c lu d in g S ustainable D e ve lo p m e n t. Energy and a g ricu ltu ra l p o licie s are th e re fo re e xcluded fro m th e inventory, unless they deal w ith en vironm enta l Issues); co n ta in eva lu a tio n c rite ria (e va lu a tio n is re q u ire d, id e a lly based o n e x p lic it c rite ria. So th e re can be no po lic y o r g u id e lin e discussion w ith o u t e va lu a tio n. Asa re su lt, m anuals fo r m u n ic ip a litie s c o n ta in in g e xp la n a tio n s o n decrees o r legal explanations o f guidelines are n o t taken in to consideration); contain recom m endations; no re c o m m e n d a tio n s (advisory papers) fro m advisory co u n cils. A lth o u g h th e y d o co n ta in c ritica l analyses (p re d o m in a n tly e x-ante e va lu ations), th e y are n o t a p p ro p ria te in an e la b o ra te ca se stu d y chapter. M o reove r, it re m ains un cle ar w h a t s c ie n tific analysis served as a basis fo r these re co m m e n d a tio n s; no articles, unless as research o r s c ie n tific analysis sum m ary. I f it is un cle ar w h e th e r th e a rticle is based on s c ie n tific analysis (i.e. based on research), it is n o t in clu d e d in th e inven tory. Ifo n th e o th e r hand it is clearly based on scie n tific analysis, th e n th e a rticle basis, in particular the research and n o t so m uch the article itself, is incorporated ; th e p o licy e va lu ations m u st be o f a high q u ality. In th e lo n g te rm, MIRA-PE w ants to be an exam ple in th e fie ld o f h ig h -q u a lity po licy eva lu ations at the service o f policym akers. T h e re fo re p u b lic a tio n s la ckin g a clear analysis fra m e w o rk w ere excluded fro m the in ve n to ry. Flowever, Flanders does n o t have a lo n g s ta n d in g tra d itio n o f p o licy e v a lu a tio n, w h ic h o b v io u s ly has repercussions on th e p ro d u c t q u a lity. C onsequently, th e in v e n to ry leaves som e ro o m fo r special cases: stu d ie s w h ic h de via te fro m th e o rd in a ry g u id e lin e s, b u t w h ic h stand o u t because o f th e ir re com m e n d a tio n s.

24 P olicy s u p p o rtin g stu d ie s w ere n o t taken in to a cco unt. T here is a fin e lin e betw een p o lic y s u p p o rtin g and p o lic y e va lu a tin g ' research th o u g h. D e p e n d in g on th e research o b je ctive, th e im p o rta n c e o f th e s u p p o rt and e v a lu a tio n e le m e n ts w ill d iffe r. D iffe re n t p o lic y studies serve d iffe re n t purposes: gain scientific insigh t in to policy processes; measure effects; inte nsify th e learning w ith in an organisation. The firs t typ e aim s a t p ro v id in g 's c ie n tific in sig h ts, th e second at a c c o u n ta b ility and th e th ird aim s a t 're fle c tio n ' o n th e po licy. The p o licy e va lu ations are g e n e ra lly a im e d at a c c o u n ta b ility, th e le g itim a tio n o f p o lic y choices, p o licy processes and p o lic y results (B ouckaert, 1992) OVERVIEW OF PUBLICATIONS We lin ke d several characteristics to each in v e n to ry p u b lic a tio n. These characteristics serve asa basis fo r fu rth e r discussion (p a rt 2). The cha racte ristic 'e n v ir o n m e n ta l to p ic re quire s a b rie f e xp la n a tio n. For each p u b lic a tio n it is in d ica te d w h ic h e n v iro n m e n ta l p ro b le m co n s titu te s th e basis o f th e evaluated policy. The fo llo w in g e n v iro n m e n ta l to p ic s are in clu d e d (in sp ire d by th e M IN A -plan 3 ( ), the Flemish E nvironm ental Policy Plan): general (p o lic y aim e d a t several to p ic s c o m b in e d o r at th e e n v iro n m e n t in ge nera l); clim ate ; w aste; e u tro p h ic a tio n ; nature; w a te r-s u rfa c e w a te ra n d de sicca tion q u a lity clu ste rin g ; photochem istry ; d e p le tio n o f th e ozo ne layer; acidifica tion; hazardous substances; soil; nuisance; fra g m e n ta tio n. MIRA-PE 2005 I fv FLEMISH ENVIRONMENTAL POLICY 2003*2005 UNDER THE MICROSCOPE F u rth e rm o re, w e have catalogued each p u b lic a tio n acco rding to evaluated 'p o lic y ty p e '. The M IN A -plan 3 classification is largely m aintained. re g io n - o r ie n te d p o lic y : a p o lic y w h ic h tackles th e e n v iro n m e n ta l p ro b le m s lin ke d to s p e cific areas. A p o lic y w h ic h enhances th e q u a lity o f specific areas o f th e e n v iro n m e n t and w h ic h is re sp o n sib le fo r th e m u tu a l c o o rd in a tio n betw een the various functions; p o lic y a im e d a t t a r g e t g r o u p s : a p o lic y w h ic h influ e n ce s th e b e h a vio u r o f stake holders w h o c o n tro l th e q u a lity o f th e e n v iro n m e n t. A po licy w h ic h stim u la te s g o v e rn m e n t cooperation and a sense o f re sponsib ility am ong the afo rem ention ed participants;

25 in te g r a te d g o v e r n m e n t p o lic y : a p o lic y a im e d at th e o p tim is a tio n o f in te rn a l co o p e ra tio n and th e c o o p e ra tio n w ith o th e r p o lic y d o m a in s and m a n a g e m e n t levels; in t e r n a t io n a l p o lic y, c o o r d in a tin g p o lic y : a p o lic y w h ic h is to fo rm a s o lid basis fo r th e policy, th e p o lic y processes and the po licy structures. M any in v e n to ry p o lic y e va lu a tio n s focu s o n o n e o r m o re s p e cific p o lic y in s tru m e n ts. W henever th is is th e case, w e have in d ica te d w h a t k in d o f in s tr u m e n t is in vo lve d. We distinguish the fo llo w in g policy instrum ents: social I com m unica tive instrum ents; econom ic instrum ents ; legal instrum ents ; s u p p o rtin g in s tru m e n ts ; d ire c t in s tru m e n ts ; enforcem ent. C ontrary to M IN A -p la n 3, e n fo rc e m e n t is n o t categorised as a legal in s tru m e n t. E n fo rce m e n t is an in s tru m e n t used by th e g o v e rn m e n t, w h ic h m ay com p rise d ire c t and all kinds o f in d ire c t p o lic y in s tru m e n ts and in th a t sense tra nscends tra d itio n a l classificatio ns. It is also a separate p o lic y p ro b le m w ith separate p o lic y eva lu a tio n research. Som e legal in s tru m e n ts (e.g. a p e rm it fo r com p a n y x) are w e ll-d e fin e d and lim ite d to th e ir fie ld o f a p p lic a tio n, o th e r legal in s tru m e n ts are a c o lle c tio n o f m easures and in s tru m e n ts (e.g. th e m a nure a ctio n plan o r th e w aste im p le m e n ta tio n plan). The latter, also called s u p p o rtin g p o licy instru m e n ts, are loosely d e fin e d and have a w id e a p p lica tio n fie ld. It isa c o lle c tiv e te rm fo r pro g ra m m e s and projects in v o lv in g charges, se n sitisa tio n, fra m e w o rk s tru c tu re etc. A p o lic y e va lu a tio n in v o lv in g a s u p p o rtin g in s tru m e n t and a s im p le legal in s tru m e n t w ill o b v io u s ly s tro n g ly d iffe r in scope and focus. The cha racte ristic 'e v a lu a tio n ty p e ' in clu d e s th e concepts ex-ante, e x-n u n c and e x-post evaluation; plan, process, and product evaluation (see c h a p te r A b o u t MIRA-PE ). Table 1 shows o u r selectio n o f p o lic y e va lu ation studies since i July 2003 (24 p u b licatio n s). The p u b lic a tio n s w ere classified a cco rd in g to e n v iro n m e n ta l to p ic, and th e n a cco rd in g to year and name.

26 Table i: O verview o f policy evaluations, published betw een i July 2003 and 30 May 2005 topic title + author) s) policy type' instrum ent type2 evaluation type General Study in preparation of the evaluation o f the federal environmentspecific product policy (2003) A. Heyerick General Europeanisation o f Flanders: a screening o f the managerial structures (2003) E. Van Reybroeck, P. Bursens General Flemish structural consultation for sustainable development (2004) K. Bachus, H. Bruyninckx, M. Poesen-Vandeputte General Border crossing Flemish environmental policy (2004) K. Geeraerts, P. Bursens, P. Leroy General Follow-up o f environmental complaints by the Environmental Inspection: an efficient instrument for high supervision? (2004) A. Monsieurs General Pilot project VLAREM supervision Klein-Brabant, Transportsector (2004) R. Segers General Linking Innovation Policyand Sustainable Development in Flanders (2004) P. Van Flumbeeck, 1. Dries, J. Larosse General The Europeanisation o f Flemish officials, members o f cabinet and politicians: opinion, expertise and attention (2005) M. Baetens, P. Bursens General Carte blanche or corset? Evaluation o f the Cooperation Agreement (2005) K.Bachus General Enforcement: the missing link in the environmental policy cycle? (2005) T. De Bruyn, K. Bachus General The Implemented sustainable development policy (2005) P. Delbaere, P. Dresselaers, N. Gouzée General Environmental policy integration in Europe (2005) European Environment Agency Waste Sector-specific implementation plans in the Flemish waste policy. Effectiveness analysis of a planning Instrument (2003) ). Gysen, K. Bachus, H. Bruyninckx, L. Van Ootegem Waste Waste prevention actions and measures in secondary schools in Flanders (2003) OVAM Waste Waste flows and waste policy in elementary schools in Flanders (2003) OVAM target group l/ec/so ex ante product international ex nunc process Integration/ coordination " ex ante process International ex nunc process Integration/ en ex nunc target group process/ product target group en ex post process integration/ coordination ex ante process/ plan international ex nunc process integration su/so ex nunc process target group en ex post plan coordination su ex post product integration ex nunc process target group su ex post product target group di/so ex post product target group di/so ex post product MIRA-PE 2005 N FLEMISH ENVIRONMENTAL POLICY UNDER THE MICROSCOPE

27 topic title + author(s) policy type' instrum ent type' evaluation type Waste Presti 4: involved in environmental protection - measuring the effects (2005) A. Gommers, K. Couderé Manure Three-tier manure policy: evaluation and future exploration (2004) M. Vervaet, L. Lauwers, S. Lenders, S. Overloop Nature Nature, how? Investigation into the social framework for nature preservation in Flanders (2004) T. Leroy, 1. Loots, P. Leroy Nature Report of the Commission on the implementation of Guideline 92/43/EC on the conservation o f natural habitats and o f wild flora and fauna (2004) European Commission Nature Nature report The condition o f nature in Flanders-figures for the policy (2005) M. Dumortier, L. De Bruyn, M. Hens, J. Peymen, A. Schneiders, T. Van Daele, W. Van Reeth, G. Weyembergh, E. Kuijken Nature Financial instruments for nature, forestand landscape (2005) B. Kayaerts Nature Region-oriented nature policy: operationalityand implementation of the Bird Guideline and Habitat Guideline (2005) A. Cliquet, G. Van Hoorick,. Lambrecht, D. Bogaert Water Local water policy about to turn: interim evaluation o f the sub-basin functioning (2005) A. Crabbé Water Regional subsidies for municipal waste water treatment plants in a European context (2005) Court of Audit target group so ex post product target group su ex nunc process target group ex ante plan region 1 ex nunc process region/ l/ec/so ex post international/ process/ target group product region/ ec ex post target group product international/ region 1 ex nunc process region/ 1 ex nunc integration process integration ec ex post product 1 region = region-oriented policy; international = international policy; integration = integrated govern m ent policy; target group = target group policy; coordination = coordinating policy 2 so = social instrum ent; I = legal instrum ent; su = supporting instrum ent; ec = econom ic instrum ent; di = direct instrum ent; en = enforcem ent 2. Discussion The in v e n to ry discussion consists o f e ig h t subsections. The typ e o f policy, th e typ e o f in s tru m e n t, th e typ e o f e va lu a tio n, th e e n v iro n m e n ta l to p ics (lin ke d to b u dget), e va lu a tio n c rite ria, th e p rin c ip a l and th e e va lu a tin g in s titu tio n s are discussed in th a t ord e r. In th e s ixth sub se ctio n w e w ill answ er general q u e stio n s raised by th e in ve n to ry, in w h ic h con clu sio n s fro m p re ce d in g paragraphs are in c lu d e d. Finally, a m a trix visualises to w h a t e x te n t th e p u b lic a tio n s p ro vid e an actual eva lu a tio n o f th e e n v iro n m e n ta l p o licy asa w h o le.

28 As th e n u m b e r o f eva lu a tio n s is lim ite d, th e discussion serves p re d o m in a n tly as an illu s tra tio n o f h o w these p o lic y e va lu a tio n s to o k place. P u b licatio n s c o n ta in in g tw o categories fo r one characteristic cou nt tw ice TYPE OF POLICY Figure i shows th e d is trib u tio n o f th e p u b lic a tio n s a cco rd in g to p o lic y type. M ost in v e n to ry p u b lic a tio n s evaluate a p o lic y sp e cifica lly aim ed a to n e o r m o re ta rg e t groups. E valuations o f c o o rd in a tin g p o licie s o ccu r less fre q u e n tly. Figure 1: D is trib u tio n o f th e in v e n to ry p u b lic a tio n s acco rding to p o lic y type 16% coordinating policy region-oriented policy 22% integrated governm ent policy 37% target group policy 16% international policy We d o n o t w a n t to m ake re c o m m e n d a tio n s w ith regard to an increase in th e n u m b e r o f eva lu a tio n s fo r ce rta in categories. All categories are e q u a lly im p o rta n t. It is up to po licym akers to m ake th e rig h t choices TYPE OF POLICY INSTRUMENT A ll eva lu a tio n s w ere classified a cco rd in g to th e typ e o f evaluated in s tru m e n t. I f an eva lu a tio n d id n o t evaluate any in s tru m e n ts, it was added to th e category 'n o t applicable. Figure 2 shows the exam ined instrum ents. Figure 2: D is trib u tio n o f th e in v e n to ry p u b lic a tio n s a cco rd in g to p o lic y in s tru m e n t MIRA-PE 2005 N FLEMISH ENVIRONMENTAL POLICY UNDER THE MICROSCOPE n ot applicable 7 6 social instrum ent 4 economic instrument enfo rcem en t 3 supporting instrument 4 5 legal instrument 2 direct instrument

29 N early o n e fo u rth o f th e p u b lic a tio n s d o n o t evaluate any in s tru m e n ts. In th e o th e r eva lu a tio n s th e re is little focus o n d ire c t in stru m e n ts. T h at is n o t su rp ris in g since, a m o n g st o th e rs, in MIRA-T (th e Flem ish State o f th e E n v iro n m e n t-re p o rt), re p o rts o n d ire c t in s tru m e n ts are p ro vid e d by th e so-called response in d ica to rs, u su ally backed up by an analysis o f th e e n v iro n m e n ta l s itu a tio n TYPE OF EVALUATION The d is trib u tio n o f th e p u b lic a tio n s a cco rd in g to typ e o f eva lu a tio n is illu s tra te d by tw o figures. Figure 3 show s h o w m any eva lu a tio n s w ere carried o u t before, d u rin g o r a fte r th e concerned policy. Figure 4 show s h o w m any p u b lic a tio n s centre a ro u n d p o lic y plans, policy processes o r policy results (plan, process and product evaluation respectively). A fte r e x a m in in g th e d is trib u tio n o f th e e n v iro n m e n ta l p o lic y e va lu a tio n s per research pe rio d (fig u re 2), w e see th a t re la tive ly fe w e x-ante eva lu a tio n s are carried o u t. This may change in th e fu tu re w h e n m o re re g u la tin g im p a cta n a lyse s (RIAs) w ill be carried o u t. As o f r January 2005 a RIA has to be p e rfo rm e d fo r a ll rules and re g u la tio n s w ith re g u la tin g effects o n citizens, th e business c o m m u n ity o r n o n -p ro fit o rg a n isa tio n s. It re m ains to be seen i f th e q u a lity o f th e RIAs w ill a llo w th e ir in clu sio n in th e in ve n to ry. We also p o in t o u t th a t th e re are im p a cta n a lyse s w h ic h are n o t p o licy e va lu ating, b u t w h ic h m ig h t as w e ll p ro vid e a fo u n d a tio n fo r fu tu re po licie s. Clearly an e x-p o st p o lic y e va lu a tio n can p ro vid e new in sig h ts fo r p o lic y d e v e lo p m e n t and th e re fo re fo r e x-ante p o lic y e va lu a tio n. How ever, th is le a rn in g e ffe c t re quire s a m in im u m o f e x-p o st p o lic y eva lu a tio n research w h ic h p rovides s u ffic ie n t g e n e ric (so n o t o n ly case-specific) in sig h ts and is in a d d itio n used fo r ex-ante eva lu a tio n s. From th is perspective Flanders s till hasa lo n g w a y to g o. Figure 3: D is trib u tio n o f th e in v e n to ry p u b lic a tio n s a cco rd in g to typ e o f eva lu a tio n (according to research period) ex-post evaluation io 4 ex-ante evaluation 10 ex-nunc evaluation Figure 4 clearly show s th a t fe w eva lu a tio n s e xa m ine th e p la n n in g phase o f th e policy. The n u m b e r o f eva lu a tio n s th a t focus o n p o lic y processes is s im ila r to th e n u m b e r fo r policy results o r products.