Leadership Can Be Learned, But How Is It Measured?

Transcription

1 Maagemet Scieces for Health NO. 8 (2008) O C C A S I O N A L PA P E R S Leadership Ca Be Leared, But How Is It Measured? How does leadership developmet cotribute to measurable chages i orgaizatioal performace, that is, behavior chages i teams ad the results they produce that cotribute to the orgaizatio s overall goals? How should we explore ad documet the performace factors ad experieces of participats to uderstad the iflueces of a leadership program? To aswer these questios, Maagemet Scieces for Health developed a comprehesive yet practical approach for evaluatig the outcomes of its leadership developmet programs with health sector participats i developig coutries. This evaluatio ivolves both trackig ad documetig cocrete evidece of chage ad icorporatig the huma face of these data to uderstad the struggles ad victories behid the results achieved. This paper describes the evaluatio methodology used ad shares the lessos leared through the experiece of the Maagemet ad Leadership (M&L) ad Leadership, Maagemet ad Sustaiability (LMS) programs. Effective maagemet ad leadership are critical aspects of well-performig health orgaizatios ad programs. Eve the availability of techically competet health professioals, adequate facilities ad supplies, ad well-desiged programs does ot assure high performace i delivery of family plaig or other health services. Good leadership ad maagemet practices are harder to assess tha other aspects of high-performig health systems but are oetheless equally importat. Oe of our major measuremet challeges, therefore, is: How do we effectively measure leadership outcomes to demostrate the tagible beefits of ivestig i leadership ad maagemet which, i tur, will heighte credibility of leadership developmet as a public health itervetio? The Maagemet ad Leadership (M&L) Program ( ) ad curret Leadership, Maagemet ad Sustaiability (LMS) Program ( ) implemeted by Maagemet Scieces for Health (MSH) were charged with measurig ad documetig the mai outcomes of their leadership developmet programs (LDPs) carried out with participats i the health sector of developig coutries. This paper describes the primary outcomes of our face-to-face ad Web-based LDPs ad explais how to measure these outcomes, with what methods ad for what purpose. The lessos leared ad coclusios are draw from our experieces as evaluators for the M&L ad LMS programs over the past eight years. Authors: Nacy LeMay is Seior Program Officer for Moitorig, Evaluatio ad Commuicatios i the Leadership, Maagemet ad Sustaiability (LMS) Program at Maagemet Scieces for Health (MSH). Aliso Ellis is the Team Leader for Moitorig, Evaluatio ad Commuicatios of the same program at MSH. M S H O C C A S I O N A L PA P E R N O. 8

2 This paper is based o a chapter prepared by the Moitorig ad Evaluatio (M&E) Uit of the M&L Program ad published by the Ceter for Creative Leadership i The Hadbook of Leadership Developmet Evaluatio (Haum et al. 2007). The iformatio preseted here is a revisio ad update of the issues ad lessos i that chapter. Our iteded audiece icludes M&E specialists ad evaluators iterested i measuremet i geeral, program maagers who may wat to kow how rigorous oe ca be about measurig leadership, other agecies providig techical assistace, ad doors. M S H l e a d e r s h i p d e v e l o p m e t p r o g r a m s Sice 2000, MSH has carried out face-to-face ad Web-based programs i 49 developig coutries to build the leadership ad maagemet capacities of health maagers ad stregthe the maagemet systems required to deliver quality health services i a sustaiable fashio. The primary cliets of these programs are maagers ad their teams from Miistries of Health ad ogovermetal orgaizatios (NGOs), icludig private volutary orgaizatios, faith-based orgaizatios, ad commuity-based orgaizatios workig i the health sector. Aother type of leadership developmet implemeted by MSH addresses goverace ad trasparecy issues for multisectoral participats such as huma resource maagemet teams, Coutry Coordiatig Mechaism teams overseeig grats from the Global Fud to Fight AIDS, TB ad Malaria, or district health committees. I all cases, the uifyig theme of our leadership developmet is helpig a team, orgaizatio, or multisectoral body achieve its ow goals o matter at what level of the health system they are workig. 1 By leadership developmet program (LDP), we mea a structured process i which health maagers ad their teams lear to apply leadig ad maagig practices (see Figure 1) to address the primary challeges they face i their uit or orgaizatio while receivig feedback ad support from program facilitators. Together the team members create a shared visio, select their desired results, ad decide how they will work to achieve the results. The maager s role is to support the team i makig a commitmet to the results ad to provide the directio to make eeded chages. The LDP ot oly teaches participats ew skills for leadig ad maagig, but also shows them a ew way of workig together that ivolves self-reflectio ad opeess to learig. MSH applied ad refied the LDP i varied settigs i the developig world ad is ow maistreamig the capacity to deliver selected products, icludig the LDP. This capacitybuildig ivolves itegratig the face-to-face LDP ito the programs of other cooperatig agecies of the US Agecy for Iteratioal Developmet (USAID), local orgaizatios, ad cosultats. For example, the ACQUIRE project maaged by EgederHealth ad the Tazaia Miistry of Health is replicatig the LDP with teams from ew facilities i Kigoma, the provice where MSH piloted the itegratio of the LDP approach ito a family plaig program supported by ACQUIRE. I Zazibar, staff from a MSH parter orgaizatio facilitated the LDP with members of district AIDS committees. The madate to maistream brigs a ew challege to the table: How do we effectively build the capacity of other orgaizatios to measure ad documet leadership outcomes? 1 I Nicaragua MSH has successfully used its approach to leadership developmet with ohealth miistries, icludig the Miistry of Educatio ad Miistry of the Family. This Occasioal Paper addresses MSH s experiece with measurig leadership developmet i the health sector. M S H O C C A S I O N A L PA P E R N O. 8 2

3 Figure 1. Leadig ad Maagig Framework scaig idetify cliet ad stakeholder eeds ad priorities recogize treds, opportuities, ad risks that affect the orgaizatio look for best practices idetify staff capacities ad costraits kow yourself, your staff, ad your orgaizatio values, stregths, ad weakesses Maagers have up-to-date, valid kowledge of their cliets, the orgaizatio, ad its cotext; they kow how their behavior affects others focusig articulate the orgaizatio s missio ad stategy idetify critical challeges lik goals with the overall orgaizatioal strategy determie key priorities for actio create a commo picture of desired results Orgaizatio s work is directed by well-defied missio, strategy, ad priorities plaig set short-term orgaizatioal goals ad performace objectives develop multi-year ad aual plas allocate adequate resources (moey, people, ad materials) aticipate ad reduce risks Orgaizatio has defied results, assiged resources, ad a operatioal pla orgaizig esure a structure that provides accoutability ad delieates authority esure that systems for huma resource maagemet, fiace, logistics, quality assurace, operatios, iformatio, ad marketig effectively support the pla stregthe work processes to implemet the pla alig staff capacities with plaed activities Orgaizatio has fuctioal structures, systems, ad processes for efficiet operatios; staff are orgaized ad aware of job resposibilities ad expectatios aligig/ mobilizig esure cogruece of values, missio, strategy, structure, systems, ad daily actios facilitate teamwork uite key stakeholders aroud a ispirig visio lik goals with rewards ad recogitio elist stakeholders to commit resources Iteral ad exteral stakeholders uderstad ad support the orgaizatio goals ad have mobilized resources to reach these goals implemetig itegrate systems ad coordiate work flow balace competig demads routiely use data for decisio-makig coordiate activities with other programs ad sectors adjust plas ad resources as circumstaces chage Activities are carried out efficietly, effectively, ad resposively ispirig match deeds to words demostrate hoesty i iteractios show trust ad cofidece i staff, ackowledge the cotributios of others provide staff with challeges, feedback, ad support be a model of creativity, iovatio, ad learig Orgaizatio displays a climate of cotiuous learig ad staff show commitmet, eve whe setbacks occur moitorig ad evaluatig moitor ad reflect o progress agaist plas provide feedback idetify eeded chages improve work processes, procedures, ad tools Orgaizatio cotiuously updates iformatio about the status of achievemets ad results, ad applies ogoig learig ad kowledge M S H O C C A S I O N A L PA P E R N O. 8

4 Leadership outcomes deped o the cotext i which the LDP is delivered ad the desired results selected by participat teams. They rage from itermediate outcomes such as chages i work climate or i maagemet systems ad processes (for example, improvig the recruitmet, developmet, retetio, ad productivity of health professioals by stregtheig the huma resource maagemet system) to log-term outcomes such as service delivery results (for example, improved quality of care, icreased utilizatio of services, or better cliet satisfactio). O u r a p p r o a c h t o l e a d e r s h i p e va l u at i o : h o w d o e s i t m e a s u r e u p? Our program philosophy states that leadership, maagemet, ad orgaizatioal developmet are ot eds i themselves. They are meas to improve the delivery of health services. I this paper we show that it is possible to measure leadership outcomes at two levels: the behavior chages i participatig teams ad the results these teams produce that cotribute to their orgaizatio s overall goals. This measuremet goes beyod the commo practice of collectig participat feedback usig a post-program evaluatio form that rates the quality of the leadership traiig. It also differs from programs that are evaluated i terms of idividual learig ad chages i practice. Some base their assessmet o idicators of idividual improvemet such as ehaced professioal kowledge. Others attempt to measure the extet to which idividual chages lead to farther-reachig outcomes, icludig orgaizatioal, system, ad societal chages. Our approach to measuremet developed through a process of trial ad error ad experimetatio with a variety of methods. Over time we have expaded our reportig focus from primarily trackig activity completio to measurig chages i cliet orgaizatios or i the populatios served by these orgaizatios. We were ispired by the work of the Populatio Leadership Program (PLP) ad the Sustaiable Leadership Iitiative (SLI) at the Public Health Istitute 2 ad the Leadership Evaluatio Advisory Group (LEAG), all based i Oaklad, Califoria. While also focused to a certai extet o idividual leadership skills ad persoal trasformatio, PLP developed the EvaluLEAD framework for evaluatig leadership outcomes at the idividual, orgaizatioal, ad commuity levels usig a combiatio of quatitative ad qualitative approaches (Grove 2002). We foud this tactic extremely useful for our particular applicatio. Give the madate of LMS to maistream ad scale up leadership developmet, we are still learig how best to measure the expasio of our leadership programs ad how best to trai other orgaizatios to moitor ad evaluate leadership outcomes M S H O C C A S I O N A L PA P E R N O. 8

5 S E C T I O N I : How Do We Develop Maagers Who Lead? We defie a maager who leads as someoe, at ay level of a orgaizatio, who applies the leadig practices show i Figure 1. T h e l e a d i g a d m a a g i g p r o c e s s Maagers ad their teams from all levels of a orgaizatio are ivited to participate i the LDP. The focus o teams rather tha idividuals is a uique feature of the leadership program. Fidigs from evaluatios coducted uder the M&L Program idicate that the compositio of the team ca affect its ability to address the challege selected ad to achieve desired results. Itact teams seem to have the best success i the log ru, ot surprisigly. A itact team is oe that has worked together o a regular basis prior to the program ad cotiues to work together after the program eds. Members of a itact team who go through the leadership developmet process ad who cotiue workig together are also more likely to be affected by the chages i social orms withi the group. More effective work habits emerge ad are mutually reiforced, thereby sustaiig the chages withi the team. Aother key igrediet is the premise that leadership ca be leared. Usig a actiolearig approach, participats lear to apply a set of leadig ad maagig practices to address their real workplace challeges over time (MSH 2005a). Whereas traditioal leadership programs ofte physically ad psychologically separate the participat from his or her work eviromet, the LDP coects the traiig to curret challeges facig participats i their workplaces. Our approach also assumes that leaders are ot bor but developed: leadig ad maagig are skills that ca be leared. Furthermore, these skills are essetial to achievig results. Our leadership programs maitai that all members of a team, regardless of their positios, ca lear to lead ad maage well i order to create a positive work climate ad achieve results. For the purposes of the LDP, leadig well meas eablig others to face challeges, achieve results, ad create the positive future that people evisio. Maagig well meas esurig that soud strategies ad approaches are i place ad resources are used effectively. The LDP approach draws upo the work of experts i leadig chage, icludig Kotter (1996) ad Heifetz ad Laurie (1997). Participats feedback has also shaped the program. Feedback from participats evaluatios ad formal program evaluatios has iflueced the terms we use to discuss leadership as well as the legth ad format of our traiig ad some of the traiig tools. A f o c u s o c h a l l e g e s The ceterpiece of the leadership developmet approach is the Challege Model (MSH 2005a), a simple yet powerful aalysis tool derived from the Performace Improvemet process (Luoma ad Voltero 2002). (See Figure 2.) Participatig teams use the model to create a shared visio of success, aalyze their local coditios, ad select a challege from their workplace that is importat ad compellig. All teams are ecouraged to select a challege i a specific techical area, such as family plaig/ reproductive health, child health, HIV& AIDS, or huma resource maagemet. The team the carries out the followig steps: measures the baselie i relatio to the challege; M S H O C C A S I O N A L PA P E R N O. 8

6 agrees o its expected measurable result; defies idicators to measure its results. idetifies mai obstacles ad their root causes that must be addressed to achieve the desired result; selects a series of priority actios to address the challege; This iformatio is traslated ito a actio pla that serves as a maagemet tool for participatig teams ad forms oe basis for evaluatig the outcomes. Figure 2. The Challege Model Missio Visio Measurable result: Priority actios Obstacles ad root causes Curret situatio: Challege: [ How will we achieve our desired result i light of the obstacles we eed to overcome? ] M S H O C C A S I O N A L PA P E R N O. 8

7 S t r at e g i e s f o r c o d u c t i g l e a d e r s h i p d e v e l o p m e t p ro g r a m s MSH facilitates leadership programs through face-to-face (o-site) ad virtual (Web-based) mechaisms. The face-to-face program is delivered to teams who most ofte come from a sigle orgaizatio or health program, 4 whereas the virtual program is delivered to teams from oe or more orgaizatios i a sigle coutry or regio. Face-to-face leadership developmet programs. The face-to-face leadership developmet program lasts from four to ie moths ad is iteded for members of preexistig teams from the public sector or private, ogovermetal, or commuity orgaizatios. Durig the program, participats atted a series of workshops facilitated by iteratioal or local MSH staff or cosultats ad the complete follow-up assigmets together at their workplaces. The program egages a cofacilitator or coach from the cliet orgaizatio to provide ogoig support to teams. Betwee the workshops, the team members orgaize meetigs durig which they discuss what they have leared ad review their progress i preparig or implemetig their actio plas. Through this process they lear to apply specific leadig ad maagig practices to address their selected challeges ad achieve their desired results. Outcomes are defied i the actio plas the teams produce durig the program. They receive regular coachig throughout the program by program facilitators (face-to-face or log-distace usig ) ad local health maagers. 4 Recet exceptios are the series of face-to-face LDPs i Guyaa delivered to several differet orgaizatios participatig i the Guyaa HIV & AIDS Reductio ad Prevetio (GHARP) Project, ad the LDP i Nepal, which ivolved more tha 25 teams from multiple sectors, icludig health, wome i developmet, forestry, ad agriculture. The Virtual Leadership Developmet Program. The Virtual Leadership Developmet Program (VLDP) is a 12- to 16-week bleded learig program. Bleded learig meas the program icludes Web-based facilitatio, course material, ad idividual exercises combied with face-to-face (o-site) team meetigs for reflectio ad shared learig amog participats. Participats from public ad private orgaizatios eroll as teams i the VLDP. Durig each module, participats carry out idividual work o the VLDP website, which icludes readig modules, case studies, ad editorials, completig exercises, ad participatig i electroic discussios. Followig this, participats covee for face-to-face meetigs with other team members at their workplaces to discuss what they leared durig the module ad to complete group work. As they move through the olie modules, teams ca commuicate with each other usig two of the website features: the Café ad the Forum. The Café is a locatio o the VLDP website where participats are ecouraged to exchage ideas o themes preseted i a threaded discussio format. The Forum is aother locatio o the website, where at the coclusio of each module a coordiator from each team describes how the group members worked together o the assigmet for the module ad what they produced. The program cotet cosists of seve modules o topics such as leadership i health istitutios, how to address a leadership challege, leadership competecies, commuicatio, ad chage maagemet. As i the face-to-face LDP, teams i the VLDP produce actio plas defiig their expected measurable results, priority actios, ad idicators. They also receive regular coachig ad feedback from program facilitators durig the program via ad postigs o the program website. M S H O C C A S I O N A L PA P E R N O. 8

8 S E C T I O N I I : Makig the Case for Measurig Leadership The assumptio that leadership ca be leared supports the idea that leadership outcomes ca be measured. Each of the elemets of leadership developmet learig materials, workshops, olie modules, feedback ad support, ad assistace to build specific skills, chage behaviors, ad work toward cocrete results ca be moitored ad documeted. The results model i Figure 3 illustrates the program priciple that measuremet of leadership ad maagemet capacity is ot a ed i itself; rather, workig o leadership ad maagemet is a meas of improvig work climate ad maagemet systems ad evetually of stregtheig health services. Our evaluatios focus o results at the output ad outcome levels ad the process used to achieve them. As i the ope systems perspective supported by the EvaluLEAD framework, our aim is to determie cotributio to results rather tha demostrate causality. Figure 3. The Leadig ad Maagig for Results Model Leadig ad maagig practices leadig maagig sca focus alig/ mobilize pla orgaize implemet improved work climate improved maagemet systems improved capacity to respod to chage Results improved services improved health outcomes ispire moitor ad evaluate L e a d e r s h i p o u t p u t s Leadership outputs are defied as the applicatio of specific leadig ad maagig practices (sca, focus, alig/mobilize, ad ispire) by participatig teams both durig ad after the leadership program. Maagers ad team members at ay level of a orgaizatio ca use these practices to address workplace challeges. The ability of the team members to apply the leadership practices i their workplace is the direct product (output) of the leadership developmet process. MSH developed a set of idicators as a selfassessmet tool for teams to moitor their behavior with respect to the four leadig practices both durig ad after a LDP (see Table 1). A score sheet of five to seve items is used to assess the fulfillmet of each idicator (see the example i Table 2 for the scaig idicator). Oly a few of our leadership programs to date have adopted or applied the idicators, sice it is usually premature to use them durig a leadership program whe teams are just learig to apply the four practices. They are most useful i assessig a team s performace before M S H O C C A S I O N A L PA P E R N O. 8 8

9 the program to determie eeds for leadership developmet or as a follow-up measure of a team s ability to cotiue applyig the practices after completio of the LDP or VLDP. To gai isights o behavior chage amog program participats ad the distict processes the teams used to achieve their results, we ofte collect iformatio o the use of the leadig practices through post-program questioaires, iterviews, ad focus group discussio with participats ad oparticipats, icludig those who report directly to the team maager ad those who do ot. While we have ot systematically tracked the leadership idicators, they still provide a valuable summary of the key actios that make up each leadership practice, which the evaluator should probe durig idividual ad group iterviews. We recogize that participat observatio would add greater depth to our iquiry o the chages i orms ad behaviors withi the group, but our ability to carry out observatio of practices i the workplace has bee limited by time ad fuds. Table 1. Output Idicators for Leadership Developmet Leadig Practice Scaig Focusig Aligig ad mobilizig Ispirig Idicator The work group ca provide valid ad relevat evidece about the ature of its iteral ad exteral eviromet, the quality ad extet of its performace, ad the resources available o best practices ad ca idetify challeges withi ad facig the team. The work group has idetified priority challeges to be addressed withi a defied time ad selected measurable actios that address barriers to achievig results. Work group resposibilities ad resources are iterally aliged ad work group goals are exterally aliged i order to address selected challeges ad meet stated objectives. Work groups are committed to the orgaizatio s missio ad to cotiuous learig, improvemet, ad iovatio. I t e r m e d i at e o u t c o m e s We track two differet itermediate outcomes defied at the team level: better work climate ad chages i maagemet systems ad processes. These are cosidered to be the ecessary precursors to improvig health services (log-term orgaizatioal outcome). Work climate is a idicatio of a team s iteral well-beig ad growth. It is defied as the prevailig workplace atmosphere that is experieced by the members of a give work group. Climate is what it feels like to work together i a group (MSH 2002). Every orgaizatio, office, ad team has a climate that affects how people behave at work, which, i tur, iflueces workers motivatio. Accordig to the Leadig ad Maagig for Results Model, work climate is a itermediate outcome that is sesitive to chage as a result of a leadership developmet process ivolvig maagers ad their teams. This relatioship ivolves two assumptios: (1) whe team members work together successfully o a shared challege, a positive work M S H O C C A S I O N A L PA P E R N O. 8

10 climate is created ad (2) work climate iflueces the achievemet of log-term outcomes. Therefore it is a suitable itermediate outcome alog the cotiuum of results. Research from the busiess ad educatio sectors has show that teams with a positive, supportive climate ted to perform well ad achieve their desired results (Golema 2000; Laschiger, Fiega, ad Shamia 2001). Better-performig teams cotribute to better orgaizatioal performace, which traslates ito improved health service delivery. Table 2. Scaig Score Sheet with Key Questios I the past three moths has the work group... Item Yes No Evidece (If Yes) 1. Reviewed ad discussed iformatio from the followig sources: Service statistics? Cliet satisfactio surveys? Commuity-based surveys? Policy-related issues or statemets? Other, please specify 2. Reviewed ad discussed the stregths, weakesses, ad/or eeds of the work group members? 3. Exchaged ideas or iformatio with exteral or iteral parters? 4. Requested iput ad participatio from all work group members durig regular meetigs? 5. Reviewed best practices related to work group goals ad objectives? Because work group climate caot be observed directly, it is estimated through the use of a questioaire measurig the perceptios of team members. I 2002, MSH developed a brief assessmet form called the Work Group Climate Assessmet (WCA), a self-scored questioaire. The items i the survey (see Table 3) are derived from the work of George Litwi ad Robert Striger, who developed the first surveys to measure climate i corporate eviromets (Litwi ad Striger 1968; Striger 2002). MSH validated the WCA i 2004 with 42 teams from differet admiistrative levels i Brazil, Mozambique, ad Guiea (Perry et al. 2005). The Striger Orgaizatioal Climate Survey was used as the stadard durig the validatio study. M S H O C C A S I O N A L PA P E R N O

11 Table 3. Items from the Work Group Climate Assessmet I feel that i my work group... How are thigs ow i your work group? Please rate each item o a scale from 1 to 5 where: 1 = ot at all 2 = to a small degree 3 = to a moderate degree 4 = to a great degree 5 = to a very great degree 1. We feel our work is importat We strive to achieve successful outcomes We pay attetio to how well we are workig together We uderstad the relevace of the job of each member i our group We have a pla that guides our activities We uderstad each other s capabilities We seek to uderstad the eeds of our cliets We take pride i our work The WCA is meat to be used before ad after a leadership program to provide comparative data o the work climate withi itact teams. To apply the questioaire, all team members (both maagerial ad staff) respod. Respodets rate how they feel about each item usig a Likert scale of 1 to 5. The scores are the tabulated across all items to produce idividuallevel composite scores ad a overall climate score for the team. Usig these scores, comparisos ca be made betwee work groups i a orgaizatio, betwee pre- ad post-itervetio assessmets of the same work group, or betwee a sigle work group ad a predetermied value of climate servig as a target goal. The secodary purpose of the WCA is to egage team members i a coversatio about their climate so that together they ca fid ways to improve it. Team members respod idividually to the survey ad afterward they are ecouraged to discuss ad act o the results together. The full WCA istrumet ad istructios (MSH 2005b) are available o MSH s website at: module=toolkit&laguage=eglish Evaluatig progress i stregtheig a particular maagemet system is ofte based o i-depth iterviews ad/or focus group discussios with the program facilitators, iteral ad exteral stakeholders, ad staff of the cliet M S H O C C A S I O N A L PA P E R N O

12 orgaizatio, as well as a review of crucial documets liked to or produced by the maagemet system uder review (for example, guidelies, operatioal plas). I some cases MOST 5 is used to detect improvemets i a sigle maagemet system or across several maagemet systems at a time. L o g - t e r m o u t c o m e s The desired outcomes of the leadership program are defied by the participats themselves ad deped o the workplace challege they choose to address durig the program. As the results model shows, MSH s leadership programs aim to cotribute to improvig health outcomes (for example, chages i the kowledge, attitudes, or practices of a target populatio or cliet group or chages i morbidity ad mortality rates). We do this by developig leaders who ca directly improve the delivery of health services. Normally we do ot measure impact-level results (health outcomes) because these are too log term for our time ad fudig costraits. 6 Whe appropriate, we do measure service delivery results. To expect chages i health services, participats i the leadership program must have a direct resposibility for health services (such as i a district health team or health facility persoel). Whe this is the case, the results of the leadership programs are measured 5 The Maagemet ad Orgaizatioal Assessmet Tool (MOST) is a self-assessmet tool that uses a participatory approach to idetify deficiet areas i a orgaizatio s maagemet systems ad defie how the weakesses will be addressed. Results of a MOST assessmet are usually used to oriet subsequet techical assistace. 6 We ecourage teams i the leadership program to select challeges that ca be accomplished i a short time (six to eight moths), which precludes workig directly toward results that show a impact o health. Oe exceptio to date is the leadership program i Aswa Goverorate, Egypt, which has succeeded i reducig the materal mortality rate over the past several years. The goverorate was able to take o such a log-term challege because they did ot have to report short-term results to a door. i terms of chages i health service delivery (for example, chages i the umber of cliets served or the quality of services). The ability to measure chage at the service delivery level is determied by several factors, icludig: the legth of time ad ature of our work with a team or orgaizatio; the challege selected by participats; whether itervetios are focused o addressig orgaizatioal challeges at the cetral level, district level, or local level; the fuctios, roles, ad resposibilities assiged to the team ivolved i the leadership program (whether cetral, regioal, or local). Measurig services is ot appropriate whe teams have ot selected a service delivery challege to address durig the program or whe the scope or time frame of the program is too limited to accomplish results at this level. I such cases, we ofte measure itermediate outcomes, such as chages i work group climate or i orgaizatioal maagemet processes ad systems, to demostrate whether or ot the teams have made progress i the directio of improvig health services. E va l u at i o ta c t i c s Our evaluatios are summative, reviewig both process ad results. The team forms the uit of aalysis because it is the focus of the faceto-face ad virtual leadership programs. We use a balaced mix of qualitative ad quatitative methods. The choice of methods depeds o the objectives of the evaluatio, the types of challeges selected by the teams ad their expected results, ad the cotet of the actio pla uder review. That said, the quatitative methods geerally iclude the use of the WCA to measure climate at the team level ad M S H O C C A S I O N A L PA P E R N O

13 the collectio of idicator data from teams to measure results accordig to their actio plas. I cases where it is possible to measure chages i health services, service delivery data (usually service statistics) are provided by the teams ad aalyzed. 7 Other methods at the service level ca iclude the use of observatio checklists to assess provider performace ad facility audits to assess cliical ifrastructure ad equipmet. For virtual programs, questioaires are set to all teams that complete the VLDP. The correspodig qualitative portio of the evaluatio aims to uderstad ad documet chages i behaviors ad processes withi the teams both durig ad after the program. We take qualitative iformatio seriously, to esure that participats have a voice i our evaluatios. Methods iclude documet reviews as well as focus group discussios ad semistructured iterviews with samples of program participats ad oparticipats. At the service level, methods ca also iclude cliet exit iterviews to measure cliet satisfactio. For virtual programs, telephoe iterviews are carried out with carefully selected iformats. All LDP ad VLDP traiig materials as well as evaluatio istrumets are traslated ito the participats primary laguage. From time to time we have ecoutered difficulties with traslatio issues, such as durig the applicatio of the WCA i Mozambique, where the majority of health staff at the peripheral level did ot speak Portuguese. I this case, a WCA facilitator iterpreted for respodets i their local laguage. Due to time ad fudig costraits, we caot carry out iterviews with all participats i a leadership program. Therefore our evaluatios most ofte use a purposive sample. Purposive samplig is a form of oprobability samplig i which respodets are selected accordig to a specific pla or purpose. Purposive samplig is differet from probability samplig, whereby each member of the populatio has a equal chace of beig sampled ad the results ca be geeralized to the sample populatio. Purposive samplig is useful for reachig a targeted sample ad whe samplig for proportioality is ot the mai cocer. The disadvatage of a purposive sample is that it is hard to kow how well the sample represets the populatio. It is importat to qualify the fidigs ad ote whether people left out of the sample might behave differetly from those who were selected. For example, i our evaluatios, teams are usually rated ad selected accordig to criteria such as the quality of their actio plas ad their adherece to the SMART 8 priciples for defiig measurable results. These criteria are iteded to esure, as much as possible, maximum variability i the performace of teams icluded i the sample. K e y q u e s t i o s w e a d d r e s s Our leadership program evaluatios are geerally based o a similar set of key questios. This allows us to sythesize the lessos leared across programs ad evaluatios. It is useful to triagulate several data sources for the same questio to verify the iformatio collected. Examples of the types of questios addressed i our evaluatios iclude: 7 While usig service statistics for measurig outcomes is ofte expediet ad ecoomical, there are also challeges from a measuremet perspective. Some of the primary challeges are outlied i Data ad Iformatio Systems: Issues for Performace Measuremet, i Perri et al. 1999, pp SMART: Specific: to avoid differig iterpretatios; Measurable: to allow moitorig ad evaluatio; Appropriate: to the problem, goals, ad strategies of the orgaizatio; Realistic: achievable, challegig, ad meaigful; ad Time boud: with a specific time period for achievig them. M S H O C C A S I O N A L PA P E R N O

14 What techical assistace approaches ad tools were used i deliverig the LDP? What other results were achieved that are urelated to addressig their challeges? What orgaizatioal challeges did the teams address through this program? What processes did participats establish to address their challeges (for example, team meetigs, aalyzig local data, cosultig iteral stakeholders)? Did teams develop actio plas to address their challeges? If so, were all plaed activities carried out? Were activities implemeted that were ot icluded i the actio pla? How did the teams ad their orgaizatios moitor their progress i addressig their challeges? To what extet did the teams ad their orgaizatios achieve their expected results? What motivated participats to achieve their desired results? What preveted them from achievig them? Did the teams cotiue to work together to address aother challege after the leadership program eded? If so, what processes were used ad how were they similar to or differet from how they worked together durig the LDP? What motivated their participatio ad commitmet after the formal program eded? To what extet was cotet from the LDP shared with staff members who did ot participate i it? I what ways did the LDP affect the performace of idividuals, teams, ad the orgaizatios as a whole? M S H O C C A S I O N A L PA P E R N O

15 S E C T I O N I I I : Lessos Leared from a Evaluatio Perspective We leared three kids of lessos i the course of this work: lessos about measurig logterm outcomes, lessos about measurig work climate, ad lessos about evaluatig the VLDP. W h at w e l e a r e d f r o m m e a s u r i g l o g - t e r m o u t c o m e s Lesso 1: The quality of participats M&E systems is a critical factor i the ability to show ad sustai results at the service delivery level. The cliet orgaizatio s M&E system determies its ability to collect ad process data ad provide accurate iformatio. The ability to track chages depeds o the quality ad accuracy of service statistics ad, as i the case of the leadership program i Aswa, Egypt, the appropriateess of the idicators selected. Teams i the Egypt program that had selected family plaig as their challege used a idicator madated by the Miistry of Health (MOH), which did ot follow iteratioally accepted guidelies for calculatig couple years of protectio. Moreover, the idicator they used did ot accurately measure the teams family plaig service results, i some cases uderreportig results, i other cases overreportig them. Lesso 2: Workig with cliets to moitor progress ad measure fial results as a itrisic part of a leadership program ca create a culture of iformatio that outlasts the program itself. The leadership program carried out by the MOH ad the M&L Program i Nampula Provice, Mozambique, from 2004 to 2005 demostrates this priciple. The program used simple maagemet ad leadership tools to assist six districts ad 11 health uits ad their commuities to improve discrete aspects of service delivery. MOH ad LDP facilitators supported the health uits i developig actio plas with measurable idicators ad feasible moitorig mechaisms. I the process the program created a culture of iformatio amog health care maagers ad providers i a settig where performace had ever bee moitored ad M&E skills were very limited. This culture of iformatio represeted a ew way of operatig at the peripheral level based o the use of data for plaig, verificatio, ad reflectio. The fial program evaluatio i 2005 showed that, for the most part, the health uits achieved surprisig results, give the low level of quality i the uits ad their extreme lack of resources. For example, health services improved i 10 of 11 health uits, resultig i shorter waitig times for patiets, improved clealiess ad hygiee, icreased biosecurity, improved quality of food, costructio of a materity waitig home, ad more accurate medical records. Furthermore, the health uits were successful i measurig their performace usig tools they had developed themselves. Table 4 shows the types of results that were moitored ad reported by health uits i this program. Prior to the program, teams from the participatig health uits had miimal skills i developig idicators ad moitorig performace. They leared to develop idicators that were appropriate to their challeges yet feasible to measure i their settig. For example, they had to fid a way to defie the criteria for clealiess, hygiee, ad biosecurity i a settig where they had few or o cleaig supplies ad the health uits were i a state of decay. While developig these idicators was difficult ad time-cosumig, participats were much more likely to take owership of the process because they had helped defie how their success would be measured. M S H O C C A S I O N A L PA P E R N O

16 Table 4. Types of Leadership Results Documeted by Selected Health Uits i Mozambique Name Health Uit Type* Challeges Reported Results, Nov Oct Mecota Health Ceter Health Ceter Type 1 Icrease percetage of births i the health facility from 25% to 35% Icrease availability of water from 2 to 12 hours/day Icreased percetage of births i the health facility from 25% to 35% Completed costructio of materity waitig home i Nov Dug but did ot complete well Namialo Health Ceter Health Ceter Type 1 Icrease average mothly percetage of cotraceptive users from 15% to 50% of quota defied by the Provicial Directorate of Health (774 users) Icreased average mothly percetage of cotraceptive users from 15% to 38% i first 9 moths of 2004 Ilha de Moçambique Health Ceter Health Ceter Type 1 Improve biosecurity Improve clealiess ad hygiee Icreased percetage of biosecurity criteria met from 16% to 83% Icreased percetage of clealiess ad hygiee criteria met from 11% to 67% Lumbo Health Ceter Health Ceter Type 2 Improve iteral ad exteral clealiess ad hygiee Improve the diet of ipatiets Icreased percetage of clealiess ad hygiee criteria met from 20% to 74% Costructed biomedical waste cotaier Paited the health ceter Built a kitche Nacala Porto Geeral Hospital Regioal Hospital Icrease umber of births i the health facility Improve quality of ipatiet care Icreased mothly average umber of births i the facility from 302 for first 6 moths of 2003 to 350 for first 6 moths of 2004 Data ot reported o ipatiet care Carapira Health Ceter Health Ceter Type 1 Reduce waitig time for pediatric visits Reduce umber of errors i ipatiet registry ad Kardex (medicatio registry) Reduced waitig time for pediatric visits by 2.5 hours Reduced errors i the ipatiet registry from 9 i Dec to 0 i Oct Reduced errors i the Kardex from 100 i Dec to 0 i Oct * A Type 1 Health Ceter offers basic primary health services ad provides ipatiet materity care for vagial deliveries; it may have a medical techicia o staff but does ot have a physicia. A Type 2 Health Ceter does ot offer ipatiet materity services ad has o techical medical staff. A regioal hospital has physicias ad offers a rage of medical ad surgical services. M S H O C C A S I O N A L PA P E R N O

17 A urse at the Ilha de Moçambique Health Ceter explaied how the ew culture of results is cotributig to better services: We did t ecourage our staff to step back from their work ad evaluate their activities before this program. We had plety of ideas, but we asked for help ad moey whe we ecoutered problems ad we ever evaluated ourselves. Now the hospital wards have a healthy spirit of competitio regardig which ward has the best results whe their activities are matched agaist the criteria that we all established. M&L coducted a evaluatio five moths after the leadership program eded. The evaluatio revealed that several teams were still usig their tools to moitor progress ad, importatly, they were able to chart their post-program performace compared to performace durig the program. Lesso 3: Not all leadership results are quatifiable. A mixed method approach is the most appropriate way to measure the results of a leadership program. Because leadership developmet is as much about the process as it is about the results, evaluators should always cosider usig a balaced mix of qualitative ad quatitative methods to collect data. Evaluatios that rely solely o quatitative data such as service delivery results may coceal importat chages i team dyamics ad their iteractios with the larger system aroud them. The PLP, LEAG, ad SLI have bee i the forefrot of developig ways to evaluate leadership developmet i the health sector. They have promoted the importace of triagulatig methods ad worked to develop a coceptual guide for assessig leadership developmet. The result is the EvaluLEAD framework, which outlies the evocative ad evidetial lies of iquiry ecessary to capture outcomes at the idividual, orgaizatioal, ad societal levels (Grove 2002 ad 2005). Our use of mixed methods was ispired by this framework. The Mozambique evaluatio provides a good example of a balaced mix of methods ad data sources, icludig: reports ad other documets related to the Health Systems Stregtheig (HSS) Program; reports from the November 2004 evaluatio workshop i Nampula Provice o health uits progress i implemetig their actio plas; reports o the progress of the MOH s Aual Operatioal Plas for 2004 ad the first trimester of 2005; observatios durig site visits i November 2004 to Carapira, Lumbo, ad Mecota health uits ad follow-up visits i April 2005 to Carapira, Lumbo, Mecota Mossuril, ad Ilha de Moçambique health uits; results of the WCAs at baselie ad after completio of the program i the Directorate of Admiistratio, Directorate of Reproductive Health, ad participatig health uits; commets from program participats i the LeaderNet 9 virtual forum Achievig Results i Low-Resource Settigs: Stories from Mozambique held i Jue 2005; 9 LeaderNet is a global commuity of practice a group of practitioers who regularly egage i sharig ad learig i a specific techical area that was developed durig the M&L Program ad cotiues to be implemeted i LMS. It operates usig both face-to-face meetigs ad through the Web, telephoe, ad fax, depedig o the eeds of its members. LeaderNet serves as a resource for maagers ad other professioals seekig to improve their leadership ad maagemet skills. Participats i all LMS programs, LDPs, VLDPs, ad semiars offered by the Global Exchage Network (GEN) for Reproductive Health, amog other programs, are ivited to joi LeaderNet: LeadigAdMaagig/LeaderNet.cfm. M S H O C C A S I O N A L PA P E R N O

18 semistructured iterviews with: selected participats i the Challeges ad Quality Programs i Nampula Provice; staff i the health uits who did ot participate i the workshops; facilitators of the Challeges ad Quality Programs; selected iformats from M&L Mozambique staff ad key maagers i the MOH; represetatives of the iteratioal door commo fud (SWAP). Aother example comes from the pilot LDP i Aswa, Egypt, carried out from 2002 to This program was delivered to 10 teams of doctors ad urses at the district ad health facility levels. Expected outcomes were defied as improved climate ad improved family plaig, preatal, ad postpartum care services. The evaluatio i 2003 relied etirely o service statistics to measure outcomes. Although sigificat improvemets were measured i may of the outcome idicators related to health services, the evaluator was ot able to explai i ay detail what had led to these chages. I 2004, a year after the program i Egypt eded, MSH evaluated the replicatio of the same program with aother set of district ad health facility teams. This time the program was implemeted by the Aswa Goverorate itself with o fiacial support from M&L ad with limited, log-distace techical assistace. For this evaluatio, a mix of qualitative ad quatitative methods was used to capture chages i outcomes. Quatitative methods were agai based o a aalysis of service statistics. Qualitative methods icluded semistructured iterviews ad focus group discussios with participats ad facilitators. These iterviews provided isight ito the processes that took place withi the district ad facility teams ad throughout the health system durig the replicatio of the leadership program. This secod evaluatio provided a much richer accout of program results, with evidece of chages i services coupled with isights to help explai how the trasformatio i service delivery was associated with the leadership developmet process. Lesso 4: Carefully selectig iformats ad creatig a safe eviromet are essetial to producig high-quality data. Whe selectig key iformats or focus group members, it is importat to keep a few poiters i mid. First, respodets should represet the characteristics of the larger team that participated i the program. For example, if the team icluded doctors, urses, couselors, ad outreach workers, the the sample of respodets should iclude these types of persoel. Secod, to promote objectivity, the program facilitators or program maager should ot choose the respodets. They should be selected istead by the evaluator accordig to criteria established i the evaluatio protocol. I additio, coductig focus group discussios with participats ad oparticipats is useful to gauge the idirect effects o leadership capacity i the two groups ad cofirm the accuracy of the resposes from those who participated i the program report. There are some risks associated with coductig focus group discussios ad iterviews at the participats workplaces. If a respodet discusses work relatioships with his or her peers or supervisors, this iformatio must be kept cofidetial. For this reaso, evaluators should choose iterview questios with care ad make sure the settig is private. Focus groups should be homogeeous, or it may be difficult to elicit cadid resposes. For example, staff ad subordiates should be grouped together i oe focus M S H O C C A S I O N A L PA P E R N O

19 group or set of focus groups, ad maagers, directors, ad supervisors should be icluded i a separate focus group. Fially, the evaluator eeds to establish a cofidetial atmosphere where there is o risk of retributio to ecourage ope discussio ad hoest iput. While focus groups ad iterviews may be recorded, the verbatim trascripts are oly for the evaluator s use ad should ot be shared with respodets peers or supervisors. Lesso 5: Teams ofte eed assistace to develop high-quality actio plas that ca serve as a evaluatio tool. Durig the leadership program, each team selects a challege such as improvig delivery of family plaig services, ad develops a actio pla to idetify the activities eeded to achieve the desired results. The team also defies idicators for measurig baselie ad fial results. Thus the actio pla liks program iputs with desired outcomes. I theory, this process provides a soud basis for measurig outcomes. I practice, however, it ca be problematic to deped o teams to supply data related to their outcomes. Sometimes the data are ot available whe it is time to evaluate progress. Except i programs that use service statistics for moitorig results, teams may ot moitor their performace usig the idicators i their actio plas. There are several reasos for this problem. I some cases, team members chage after the program eds. This is especially commo i the public sector. May teams do carry out their plas but may moitor performace usig orgaizatioal-level idicators rather tha the idicators i their actio plas. I other cases, the team does ot itegrate its actio pla ito the orgaizatio s aual pla, so there may be iadequate resources to moitor usig actio pla idicators. Sometimes, key stakeholders i the orgaizatio are ot supportive, so teams either do ot fully carry out their actio plas or are ot motivated to moitor their performace. Fially, while some teams produce actio plas that are writte logically ad clearly with well-distributed resposibilities, well-defied idicators, SMART expected results, ad appropriate data sources i other cases, it is uclear how activities i the pla will lead to achievig the desired results. A well-developed actio pla is more likely to be implemeted ad its idicators used tha a poorly defied pla. Our prerequisite for usig a actio pla as the basis for a evaluatio is esurig that the pla meets the SMART criteria. To achieve this, the leadership programs have egaged M&E Uit staff to review ad provide feedback to participats o their actio plas, a feature of the VLDP that is yieldig icreasigly better results. The purpose of the review is to check that (1) the selected challege is immediately actioable ad will cotribute to the iteded results, (2) the activities logically address the challege, ad (3) the proposed idicators are appropriate to measure progress. To make sure the pla meets the SMART criteria, teams are give guidace usig the followig questios: Are the missio ad visio of the team clear, especially as they relate to the orgaizatio s overall missio? Is the challege selected by the team clear? Are the desired results defied by the team clear ad achievable withi six to ie moths? Are activities logically related to the desired results? Are measurable idicators defied? Is a time lie or time frame for implemetatio idicated? M S H O C C A S I O N A L PA P E R N O

20 Are huma ad fiacial resources idicated i the pla ad are they available? We have also developed materials o M&E for facilitators to use durig the leadership program to support those resposible for providig M&E guidace to participatig teams. Please see files/m_e_guide_to_vldp_ad_ldp_fial_ 2007.doc W h at w e l e a r e d f ro m m e a s u r i g w o r k c l i m at e MSH has used the WCA to measure work climate ad climate chage durig most faceto-face leadership programs ad all VLDPs coducted to date. Based o this experiece, we offer several lessos leared about the use of the tool ad the quality of the data it produces. Lesso 6: Improvemets i climate caot be attributed oly to the effects of the LDP or VLDP due to iterveig variables ad the log time frame of the LDP. Experiece has show that as teams lear to use leadership ad maagemet practices to address their challeges, work climate teds to improve. This is true for teams from NGOs ad the cetral, provicial, ad local levels of the public health sector. Nevertheless, there are limits to demostratig chages i climate ad attributig them to the itervetio. The log time frame of the LDP (up to a year) meas that other evets, such as political or iteral orgaizatioal chages, ca impact climate for better or worse, despite the effects of the LDP. I the ideal, our evaluatios would use a cotrol group; however, we caot radomly assig participats to such a group. Participatio i a leadership program is ot idiscrimiate ad teams are ot uiformed; teams that participate are either self-selected or are recommeded by a orgaizatioal executive or admiistrator or a door. Lesso 7: For programmatic ad evaluatio purposes, it is ideal to recruit itact teams for the LDP ad VLDP. The LDP tries to recruit itact teams. A itact team is oe whose members have worked together before begiig the LDP ad will cotiue to work together after the program eds (although all members of the team do ot have to be i the same locatio). Itact teams are more likely to carry out their actio plas, ad the behaviors they adopt durig the program ted to be sustaied withi the group afterwards. They are also more resposive to the WCA survey ad easier to follow after the program eds. Teams that are formed artificially for the purpose of participatig i a LDP (for example, a group of supervisors from various levels of a public- or private-sector orgaizatio) have more difficulty i implemetig their plas ad measurig their progress after the program eds. These teams are ofte made up of idividuals from geographically or admiistratively dispersed groups. They may work well together durig the program, but whe they retur to their ormal routies after its completio, they may o loger be motivated to work together as a team because they are ot, i fact, a team. Teams that disitegrate after the program eds are usually lost to follow-up, which egatively affects our ability to measure leadership outcomes. All members of a work group must fill out the WCA for evaluators to obtai a valid measure of work group climate. Ad because itact teams have a history of workig together, they ca respod to the WCA items i a meaigful way. Thus, ot oly for effective program desig but also for evaluatio purposes, it is preferable to recruit itact teams for leadership programs. Despite the beefits of erollig itact teams, ew kids of teams have emerged i the LDP ad VLDP i respose to the eed to stregthe M S H O C C A S I O N A L PA P E R N O