Leadership Can Be Learned, But How Is It Measured?

Maagemet Scieces for Health NO. 8 (2008) O C C A S I O N A L PA P E R S Leadership Ca Be Leared, But How Is It Measured? How does leadership developmet cotribute to measurable chages i orgaizatioal performace, that is, behavior chages i teams ad the results they produce that cotribute to the orgaizatio s overall goals? How should we explore ad documet the performace factors ad experieces of participats to uderstad the iflueces of a leadership program? To aswer these questios, Maagemet Scieces for Health developed a comprehesive yet practical approach for evaluatig the outcomes of its leadership developmet programs with health sector participats i developig coutries. This evaluatio ivolves both trackig ad documetig cocrete evidece of chage ad icorporatig the huma face of these data to uderstad the struggles ad victories behid the results achieved. This paper describes the evaluatio methodology used ad shares the lessos leared through the experiece of the Maagemet ad Leadership (M&L) ad Leadership, Maagemet ad Sustaiability (LMS) programs. Effective maagemet ad leadership are critical aspects of well-performig health orgaizatios ad programs. Eve the availability of techically competet health professioals, adequate facilities ad supplies, ad well-desiged programs does ot assure high performace i delivery of family plaig or other health services. Good leadership ad maagemet practices are harder to assess tha other aspects of high-performig health systems but are oetheless equally importat. Oe of our major measuremet challeges, therefore, is: How do we effectively measure leadership outcomes to demostrate the tagible beefits of ivestig i leadership ad maagemet which, i tur, will heighte credibility of leadership developmet as a public health itervetio? The Maagemet ad Leadership (M&L) Program (2000 05) ad curret Leadership, Maagemet ad Sustaiability (LMS) Program (2005 10) implemeted by Maagemet Scieces for Health (MSH) were charged with measurig ad documetig the mai outcomes of their leadership developmet programs (LDPs) carried out with participats i the health sector of developig coutries. This paper describes the primary outcomes of our face-to-face ad Web-based LDPs ad explais how to measure these outcomes, with what methods ad for what purpose. The lessos leared ad coclusios are draw from our experieces as evaluators for the M&L ad LMS programs over the past eight years. Authors: Nacy LeMay is Seior Program Officer for Moitorig, Evaluatio ad Commuicatios i the Leadership, Maagemet ad Sustaiability (LMS) Program at Maagemet Scieces for Health (MSH). Aliso Ellis is the Team Leader for Moitorig, Evaluatio ad Commuicatios of the same program at MSH. M S H O C C A S I O N A L PA P E R N O. 8

This paper is based o a chapter prepared by the Moitorig ad Evaluatio (M&E) Uit of the M&L Program ad published by the Ceter for Creative Leadership i The Hadbook of Leadership Developmet Evaluatio (Haum et al. 2007). The iformatio preseted here is a revisio ad update of the issues ad lessos i that chapter. Our iteded audiece icludes M&E specialists ad evaluators iterested i measuremet i geeral, program maagers who may wat to kow how rigorous oe ca be about measurig leadership, other agecies providig techical assistace, ad doors. M S H l e a d e r s h i p d e v e l o p m e t p r o g r a m s Sice 2000, MSH has carried out face-to-face ad Web-based programs i 49 developig coutries to build the leadership ad maagemet capacities of health maagers ad stregthe the maagemet systems required to deliver quality health services i a sustaiable fashio. The primary cliets of these programs are maagers ad their teams from Miistries of Health ad ogovermetal orgaizatios (NGOs), icludig private volutary orgaizatios, faith-based orgaizatios, ad commuity-based orgaizatios workig i the health sector. Aother type of leadership developmet implemeted by MSH addresses goverace ad trasparecy issues for multisectoral participats such as huma resource maagemet teams, Coutry Coordiatig Mechaism teams overseeig grats from the Global Fud to Fight AIDS, TB ad Malaria, or district health committees. I all cases, the uifyig theme of our leadership developmet is helpig a team, orgaizatio, or multisectoral body achieve its ow goals o matter at what level of the health system they are workig. 1 By leadership developmet program (LDP), we mea a structured process i which health maagers ad their teams lear to apply leadig ad maagig practices (see Figure 1) to address the primary challeges they face i their uit or orgaizatio while receivig feedback ad support from program facilitators. Together the team members create a shared visio, select their desired results, ad decide how they will work to achieve the results. The maager s role is to support the team i makig a commitmet to the results ad to provide the directio to make eeded chages. The LDP ot oly teaches participats ew skills for leadig ad maagig, but also shows them a ew way of workig together that ivolves self-reflectio ad opeess to learig. MSH applied ad refied the LDP i varied settigs i the developig world ad is ow maistreamig the capacity to deliver selected products, icludig the LDP. This capacitybuildig ivolves itegratig the face-to-face LDP ito the programs of other cooperatig agecies of the US Agecy for Iteratioal Developmet (USAID), local orgaizatios, ad cosultats. For example, the ACQUIRE project maaged by EgederHealth ad the Tazaia Miistry of Health is replicatig the LDP with teams from ew facilities i Kigoma, the provice where MSH piloted the itegratio of the LDP approach ito a family plaig program supported by ACQUIRE. I Zazibar, staff from a MSH parter orgaizatio facilitated the LDP with members of district AIDS committees. The madate to maistream brigs a ew challege to the table: How do we effectively build the capacity of other orgaizatios to measure ad documet leadership outcomes? 1 I Nicaragua MSH has successfully used its approach to leadership developmet with ohealth miistries, icludig the Miistry of Educatio ad Miistry of the Family. This Occasioal Paper addresses MSH s experiece with measurig leadership developmet i the health sector. M S H O C C A S I O N A L PA P E R N O. 8 2

Figure 1. Leadig ad Maagig Framework scaig idetify cliet ad stakeholder eeds ad priorities recogize treds, opportuities, ad risks that affect the orgaizatio look for best practices idetify staff capacities ad costraits kow yourself, your staff, ad your orgaizatio values, stregths, ad weakesses Maagers have up-to-date, valid kowledge of their cliets, the orgaizatio, ad its cotext; they kow how their behavior affects others focusig articulate the orgaizatio s missio ad stategy idetify critical challeges lik goals with the overall orgaizatioal strategy determie key priorities for actio create a commo picture of desired results Orgaizatio s work is directed by well-defied missio, strategy, ad priorities plaig set short-term orgaizatioal goals ad performace objectives develop multi-year ad aual plas allocate adequate resources (moey, people, ad materials) aticipate ad reduce risks Orgaizatio has defied results, assiged resources, ad a operatioal pla orgaizig esure a structure that provides accoutability ad delieates authority esure that systems for huma resource maagemet, fiace, logistics, quality assurace, operatios, iformatio, ad marketig effectively support the pla stregthe work processes to implemet the pla alig staff capacities with plaed activities Orgaizatio has fuctioal structures, systems, ad processes for efficiet operatios; staff are orgaized ad aware of job resposibilities ad expectatios aligig/ mobilizig esure cogruece of values, missio, strategy, structure, systems, ad daily actios facilitate teamwork uite key stakeholders aroud a ispirig visio lik goals with rewards ad recogitio elist stakeholders to commit resources Iteral ad exteral stakeholders uderstad ad support the orgaizatio goals ad have mobilized resources to reach these goals implemetig itegrate systems ad coordiate work flow balace competig demads routiely use data for decisio-makig coordiate activities with other programs ad sectors adjust plas ad resources as circumstaces chage Activities are carried out efficietly, effectively, ad resposively ispirig match deeds to words demostrate hoesty i iteractios show trust ad cofidece i staff, ackowledge the cotributios of others provide staff with challeges, feedback, ad support be a model of creativity, iovatio, ad learig Orgaizatio displays a climate of cotiuous learig ad staff show commitmet, eve whe setbacks occur moitorig ad evaluatig moitor ad reflect o progress agaist plas provide feedback idetify eeded chages improve work processes, procedures, ad tools Orgaizatio cotiuously updates iformatio about the status of achievemets ad results, ad applies ogoig learig ad kowledge M S H O C C A S I O N A L PA P E R N O. 8

Leadership outcomes deped o the cotext i which the LDP is delivered ad the desired results selected by participat teams. They rage from itermediate outcomes such as chages i work climate or i maagemet systems ad processes (for example, improvig the recruitmet, developmet, retetio, ad productivity of health professioals by stregtheig the huma resource maagemet system) to log-term outcomes such as service delivery results (for example, improved quality of care, icreased utilizatio of services, or better cliet satisfactio). O u r a p p r o a c h t o l e a d e r s h i p e va l u at i o : h o w d o e s i t m e a s u r e u p? Our program philosophy states that leadership, maagemet, ad orgaizatioal developmet are ot eds i themselves. They are meas to improve the delivery of health services. I this paper we show that it is possible to measure leadership outcomes at two levels: the behavior chages i participatig teams ad the results these teams produce that cotribute to their orgaizatio s overall goals. This measuremet goes beyod the commo practice of collectig participat feedback usig a post-program evaluatio form that rates the quality of the leadership traiig. It also differs from programs that are evaluated i terms of idividual learig ad chages i practice. Some base their assessmet o idicators of idividual improvemet such as ehaced professioal kowledge. Others attempt to measure the extet to which idividual chages lead to farther-reachig outcomes, icludig orgaizatioal, system, ad societal chages. Our approach to measuremet developed through a process of trial ad error ad experimetatio with a variety of methods. Over time we have expaded our reportig focus from primarily trackig activity completio to measurig chages i cliet orgaizatios or i the populatios served by these orgaizatios. We were ispired by the work of the Populatio Leadership Program (PLP) ad the Sustaiable Leadership Iitiative (SLI) at the Public Health Istitute 2 ad the Leadership Evaluatio Advisory Group (LEAG), all based i Oaklad, Califoria. While also focused to a certai extet o idividual leadership skills ad persoal trasformatio, PLP developed the EvaluLEAD framework for evaluatig leadership outcomes at the idividual, orgaizatioal, ad commuity levels usig a combiatio of quatitative ad qualitative approaches (Grove 2002). We foud this tactic extremely useful for our particular applicatio. Give the madate of LMS to maistream ad scale up leadership developmet, we are still learig how best to measure the expasio of our leadership programs ad how best to trai other orgaizatios to moitor ad evaluate leadership outcomes. 2 http://www.phi.org/ http://www.leadershiplearig.org/author/leadershipevaluatio-advisory-group-members M S H O C C A S I O N A L PA P E R N O. 8

S E C T I O N I : How Do We Develop Maagers Who Lead? We defie a maager who leads as someoe, at ay level of a orgaizatio, who applies the leadig practices show i Figure 1. T h e l e a d i g a d m a a g i g p r o c e s s Maagers ad their teams from all levels of a orgaizatio are ivited to participate i the LDP. The focus o teams rather tha idividuals is a uique feature of the leadership program. Fidigs from evaluatios coducted uder the M&L Program idicate that the compositio of the team ca affect its ability to address the challege selected ad to achieve desired results. Itact teams seem to have the best success i the log ru, ot surprisigly. A itact team is oe that has worked together o a regular basis prior to the program ad cotiues to work together after the program eds. Members of a itact team who go through the leadership developmet process ad who cotiue workig together are also more likely to be affected by the chages i social orms withi the group. More effective work habits emerge ad are mutually reiforced, thereby sustaiig the chages withi the team. Aother key igrediet is the premise that leadership ca be leared. Usig a actiolearig approach, participats lear to apply a set of leadig ad maagig practices to address their real workplace challeges over time (MSH 2005a). Whereas traditioal leadership programs ofte physically ad psychologically separate the participat from his or her work eviromet, the LDP coects the traiig to curret challeges facig participats i their workplaces. Our approach also assumes that leaders are ot bor but developed: leadig ad maagig are skills that ca be leared. Furthermore, these skills are essetial to achievig results. Our leadership programs maitai that all members of a team, regardless of their positios, ca lear to lead ad maage well i order to create a positive work climate ad achieve results. For the purposes of the LDP, leadig well meas eablig others to face challeges, achieve results, ad create the positive future that people evisio. Maagig well meas esurig that soud strategies ad approaches are i place ad resources are used effectively. The LDP approach draws upo the work of experts i leadig chage, icludig Kotter (1996) ad Heifetz ad Laurie (1997). Participats feedback has also shaped the program. Feedback from participats evaluatios ad formal program evaluatios has iflueced the terms we use to discuss leadership as well as the legth ad format of our traiig ad some of the traiig tools. A f o c u s o c h a l l e g e s The ceterpiece of the leadership developmet approach is the Challege Model (MSH 2005a), a simple yet powerful aalysis tool derived from the Performace Improvemet process (Luoma ad Voltero 2002). (See Figure 2.) Participatig teams use the model to create a shared visio of success, aalyze their local coditios, ad select a challege from their workplace that is importat ad compellig. All teams are ecouraged to select a challege i a specific techical area, such as family plaig/ reproductive health, child health, HIV& AIDS, or huma resource maagemet. The team the carries out the followig steps: measures the baselie i relatio to the challege; M S H O C C A S I O N A L PA P E R N O. 8

agrees o its expected measurable result; defies idicators to measure its results. idetifies mai obstacles ad their root causes that must be addressed to achieve the desired result; selects a series of priority actios to address the challege; This iformatio is traslated ito a actio pla that serves as a maagemet tool for participatig teams ad forms oe basis for evaluatig the outcomes. Figure 2. The Challege Model Missio Visio Measurable result: Priority actios Obstacles ad root causes Curret situatio: Challege: [ How will we achieve our desired result i light of the obstacles we eed to overcome? ] M S H O C C A S I O N A L PA P E R N O. 8

S t r at e g i e s f o r c o d u c t i g l e a d e r s h i p d e v e l o p m e t p ro g r a m s MSH facilitates leadership programs through face-to-face (o-site) ad virtual (Web-based) mechaisms. The face-to-face program is delivered to teams who most ofte come from a sigle orgaizatio or health program, 4 whereas the virtual program is delivered to teams from oe or more orgaizatios i a sigle coutry or regio. Face-to-face leadership developmet programs. The face-to-face leadership developmet program lasts from four to ie moths ad is iteded for members of preexistig teams from the public sector or private, ogovermetal, or commuity orgaizatios. Durig the program, participats atted a series of workshops facilitated by iteratioal or local MSH staff or cosultats ad the complete follow-up assigmets together at their workplaces. The program egages a cofacilitator or coach from the cliet orgaizatio to provide ogoig support to teams. Betwee the workshops, the team members orgaize meetigs durig which they discuss what they have leared ad review their progress i preparig or implemetig their actio plas. Through this process they lear to apply specific leadig ad maagig practices to address their selected challeges ad achieve their desired results. Outcomes are defied i the actio plas the teams produce durig the program. They receive regular coachig throughout the program by program facilitators (face-to-face or log-distace usig e-mail) ad local health maagers. 4 Recet exceptios are the series of face-to-face LDPs i Guyaa delivered to several differet orgaizatios participatig i the Guyaa HIV & AIDS Reductio ad Prevetio (GHARP) Project, ad the LDP i Nepal, which ivolved more tha 25 teams from multiple sectors, icludig health, wome i developmet, forestry, ad agriculture. The Virtual Leadership Developmet Program. The Virtual Leadership Developmet Program (VLDP) is a 12- to 16-week bleded learig program. Bleded learig meas the program icludes Web-based facilitatio, course material, ad idividual exercises combied with face-to-face (o-site) team meetigs for reflectio ad shared learig amog participats. Participats from public ad private orgaizatios eroll as teams i the VLDP. Durig each module, participats carry out idividual work o the VLDP website, which icludes readig modules, case studies, ad editorials, completig exercises, ad participatig i electroic discussios. Followig this, participats covee for face-to-face meetigs with other team members at their workplaces to discuss what they leared durig the module ad to complete group work. As they move through the olie modules, teams ca commuicate with each other usig two of the website features: the Café ad the Forum. The Café is a locatio o the VLDP website where participats are ecouraged to exchage ideas o themes preseted i a threaded discussio format. The Forum is aother locatio o the website, where at the coclusio of each module a coordiator from each team describes how the group members worked together o the assigmet for the module ad what they produced. The program cotet cosists of seve modules o topics such as leadership i health istitutios, how to address a leadership challege, leadership competecies, commuicatio, ad chage maagemet. As i the face-to-face LDP, teams i the VLDP produce actio plas defiig their expected measurable results, priority actios, ad idicators. They also receive regular coachig ad feedback from program facilitators durig the program via e-mail ad postigs o the program website. M S H O C C A S I O N A L PA P E R N O. 8

S E C T I O N I I : Makig the Case for Measurig Leadership The assumptio that leadership ca be leared supports the idea that leadership outcomes ca be measured. Each of the elemets of leadership developmet learig materials, workshops, olie modules, feedback ad support, ad assistace to build specific skills, chage behaviors, ad work toward cocrete results ca be moitored ad documeted. The results model i Figure 3 illustrates the program priciple that measuremet of leadership ad maagemet capacity is ot a ed i itself; rather, workig o leadership ad maagemet is a meas of improvig work climate ad maagemet systems ad evetually of stregtheig health services. Our evaluatios focus o results at the output ad outcome levels ad the process used to achieve them. As i the ope systems perspective supported by the EvaluLEAD framework, our aim is to determie cotributio to results rather tha demostrate causality. Figure 3. The Leadig ad Maagig for Results Model Leadig ad maagig practices leadig maagig sca focus alig/ mobilize pla orgaize implemet improved work climate improved maagemet systems improved capacity to respod to chage Results improved services improved health outcomes ispire moitor ad evaluate L e a d e r s h i p o u t p u t s Leadership outputs are defied as the applicatio of specific leadig ad maagig practices (sca, focus, alig/mobilize, ad ispire) by participatig teams both durig ad after the leadership program. Maagers ad team members at ay level of a orgaizatio ca use these practices to address workplace challeges. The ability of the team members to apply the leadership practices i their workplace is the direct product (output) of the leadership developmet process. MSH developed a set of idicators as a selfassessmet tool for teams to moitor their behavior with respect to the four leadig practices both durig ad after a LDP (see Table 1). A score sheet of five to seve items is used to assess the fulfillmet of each idicator (see the example i Table 2 for the scaig idicator). Oly a few of our leadership programs to date have adopted or applied the idicators, sice it is usually premature to use them durig a leadership program whe teams are just learig to apply the four practices. They are most useful i assessig a team s performace before M S H O C C A S I O N A L PA P E R N O. 8 8

the program to determie eeds for leadership developmet or as a follow-up measure of a team s ability to cotiue applyig the practices after completio of the LDP or VLDP. To gai isights o behavior chage amog program participats ad the distict processes the teams used to achieve their results, we ofte collect iformatio o the use of the leadig practices through post-program questioaires, iterviews, ad focus group discussio with participats ad oparticipats, icludig those who report directly to the team maager ad those who do ot. While we have ot systematically tracked the leadership idicators, they still provide a valuable summary of the key actios that make up each leadership practice, which the evaluator should probe durig idividual ad group iterviews. We recogize that participat observatio would add greater depth to our iquiry o the chages i orms ad behaviors withi the group, but our ability to carry out observatio of practices i the workplace has bee limited by time ad fuds. Table 1. Output Idicators for Leadership Developmet Leadig Practice Scaig Focusig Aligig ad mobilizig Ispirig Idicator The work group ca provide valid ad relevat evidece about the ature of its iteral ad exteral eviromet, the quality ad extet of its performace, ad the resources available o best practices ad ca idetify challeges withi ad facig the team. The work group has idetified priority challeges to be addressed withi a defied time ad selected measurable actios that address barriers to achievig results. Work group resposibilities ad resources are iterally aliged ad work group goals are exterally aliged i order to address selected challeges ad meet stated objectives. Work groups are committed to the orgaizatio s missio ad to cotiuous learig, improvemet, ad iovatio. I t e r m e d i at e o u t c o m e s We track two differet itermediate outcomes defied at the team level: better work climate ad chages i maagemet systems ad processes. These are cosidered to be the ecessary precursors to improvig health services (log-term orgaizatioal outcome). Work climate is a idicatio of a team s iteral well-beig ad growth. It is defied as the prevailig workplace atmosphere that is experieced by the members of a give work group. Climate is what it feels like to work together i a group (MSH 2002). Every orgaizatio, office, ad team has a climate that affects how people behave at work, which, i tur, iflueces workers motivatio. Accordig to the Leadig ad Maagig for Results Model, work climate is a itermediate outcome that is sesitive to chage as a result of a leadership developmet process ivolvig maagers ad their teams. This relatioship ivolves two assumptios: (1) whe team members work together successfully o a shared challege, a positive work M S H O C C A S I O N A L PA P E R N O. 8

climate is created ad (2) work climate iflueces the achievemet of log-term outcomes. Therefore it is a suitable itermediate outcome alog the cotiuum of results. Research from the busiess ad educatio sectors has show that teams with a positive, supportive climate ted to perform well ad achieve their desired results (Golema 2000; Laschiger, Fiega, ad Shamia 2001). Better-performig teams cotribute to better orgaizatioal performace, which traslates ito improved health service delivery. Table 2. Scaig Score Sheet with Key Questios I the past three moths has the work group... Item Yes No Evidece (If Yes) 1. Reviewed ad discussed iformatio from the followig sources: Service statistics? Cliet satisfactio surveys? Commuity-based surveys? Policy-related issues or statemets? Other, please specify 2. Reviewed ad discussed the stregths, weakesses, ad/or eeds of the work group members? 3. Exchaged ideas or iformatio with exteral or iteral parters? 4. Requested iput ad participatio from all work group members durig regular meetigs? 5. Reviewed best practices related to work group goals ad objectives? Because work group climate caot be observed directly, it is estimated through the use of a questioaire measurig the perceptios of team members. I 2002, MSH developed a brief assessmet form called the Work Group Climate Assessmet (WCA), a self-scored questioaire. The items i the survey (see Table 3) are derived from the work of George Litwi ad Robert Striger, who developed the first surveys to measure climate i corporate eviromets (Litwi ad Striger 1968; Striger 2002). MSH validated the WCA i 2004 with 42 teams from differet admiistrative levels i Brazil, Mozambique, ad Guiea (Perry et al. 2005). The Striger Orgaizatioal Climate Survey was used as the stadard durig the validatio study. M S H O C C A S I O N A L PA P E R N O. 8 1 0

Table 3. Items from the Work Group Climate Assessmet I feel that i my work group... How are thigs ow i your work group? Please rate each item o a scale from 1 to 5 where: 1 = ot at all 2 = to a small degree 3 = to a moderate degree 4 = to a great degree 5 = to a very great degree 1. We feel our work is importat. 1 2 3 4 5 2. We strive to achieve successful outcomes. 1 2 3 4 5 3. We pay attetio to how well we are workig together. 1 2 3 4 5 4. We uderstad the relevace of the job of each member i our group. 1 2 3 4 5 5. We have a pla that guides our activities. 1 2 3 4 5 6. We uderstad each other s capabilities. 1 2 3 4 5 7. We seek to uderstad the eeds of our cliets. 1 2 3 4 5 8. We take pride i our work. 1 2 3 4 5 The WCA is meat to be used before ad after a leadership program to provide comparative data o the work climate withi itact teams. To apply the questioaire, all team members (both maagerial ad staff) respod. Respodets rate how they feel about each item usig a Likert scale of 1 to 5. The scores are the tabulated across all items to produce idividuallevel composite scores ad a overall climate score for the team. Usig these scores, comparisos ca be made betwee work groups i a orgaizatio, betwee pre- ad post-itervetio assessmets of the same work group, or betwee a sigle work group ad a predetermied value of climate servig as a target goal. The secodary purpose of the WCA is to egage team members i a coversatio about their climate so that together they ca fid ways to improve it. Team members respod idividually to the survey ad afterward they are ecouraged to discuss ad act o the results together. The full WCA istrumet ad istructios (MSH 2005b) are available o MSH s website at: http://erc.msh.org/maipage.cfm?file=96.9htm& module=toolkit&laguage=eglish Evaluatig progress i stregtheig a particular maagemet system is ofte based o i-depth iterviews ad/or focus group discussios with the program facilitators, iteral ad exteral stakeholders, ad staff of the cliet M S H O C C A S I O N A L PA P E R N O. 8 1 1

orgaizatio, as well as a review of crucial documets liked to or produced by the maagemet system uder review (for example, guidelies, operatioal plas). I some cases MOST 5 is used to detect improvemets i a sigle maagemet system or across several maagemet systems at a time. L o g - t e r m o u t c o m e s The desired outcomes of the leadership program are defied by the participats themselves ad deped o the workplace challege they choose to address durig the program. As the results model shows, MSH s leadership programs aim to cotribute to improvig health outcomes (for example, chages i the kowledge, attitudes, or practices of a target populatio or cliet group or chages i morbidity ad mortality rates). We do this by developig leaders who ca directly improve the delivery of health services. Normally we do ot measure impact-level results (health outcomes) because these are too log term for our time ad fudig costraits. 6 Whe appropriate, we do measure service delivery results. To expect chages i health services, participats i the leadership program must have a direct resposibility for health services (such as i a district health team or health facility persoel). Whe this is the case, the results of the leadership programs are measured 5 The Maagemet ad Orgaizatioal Assessmet Tool (MOST) is a self-assessmet tool that uses a participatory approach to idetify deficiet areas i a orgaizatio s maagemet systems ad defie how the weakesses will be addressed. Results of a MOST assessmet are usually used to oriet subsequet techical assistace. 6 We ecourage teams i the leadership program to select challeges that ca be accomplished i a short time (six to eight moths), which precludes workig directly toward results that show a impact o health. Oe exceptio to date is the leadership program i Aswa Goverorate, Egypt, which has succeeded i reducig the materal mortality rate over the past several years. The goverorate was able to take o such a log-term challege because they did ot have to report short-term results to a door. i terms of chages i health service delivery (for example, chages i the umber of cliets served or the quality of services). The ability to measure chage at the service delivery level is determied by several factors, icludig: the legth of time ad ature of our work with a team or orgaizatio; the challege selected by participats; whether itervetios are focused o addressig orgaizatioal challeges at the cetral level, district level, or local level; the fuctios, roles, ad resposibilities assiged to the team ivolved i the leadership program (whether cetral, regioal, or local). Measurig services is ot appropriate whe teams have ot selected a service delivery challege to address durig the program or whe the scope or time frame of the program is too limited to accomplish results at this level. I such cases, we ofte measure itermediate outcomes, such as chages i work group climate or i orgaizatioal maagemet processes ad systems, to demostrate whether or ot the teams have made progress i the directio of improvig health services. E va l u at i o ta c t i c s Our evaluatios are summative, reviewig both process ad results. The team forms the uit of aalysis because it is the focus of the faceto-face ad virtual leadership programs. We use a balaced mix of qualitative ad quatitative methods. The choice of methods depeds o the objectives of the evaluatio, the types of challeges selected by the teams ad their expected results, ad the cotet of the actio pla uder review. That said, the quatitative methods geerally iclude the use of the WCA to measure climate at the team level ad M S H O C C A S I O N A L PA P E R N O. 8 1 2

the collectio of idicator data from teams to measure results accordig to their actio plas. I cases where it is possible to measure chages i health services, service delivery data (usually service statistics) are provided by the teams ad aalyzed. 7 Other methods at the service level ca iclude the use of observatio checklists to assess provider performace ad facility audits to assess cliical ifrastructure ad equipmet. For virtual programs, e-mail questioaires are set to all teams that complete the VLDP. The correspodig qualitative portio of the evaluatio aims to uderstad ad documet chages i behaviors ad processes withi the teams both durig ad after the program. We take qualitative iformatio seriously, to esure that participats have a voice i our evaluatios. Methods iclude documet reviews as well as focus group discussios ad semistructured iterviews with samples of program participats ad oparticipats. At the service level, methods ca also iclude cliet exit iterviews to measure cliet satisfactio. For virtual programs, telephoe iterviews are carried out with carefully selected iformats. All LDP ad VLDP traiig materials as well as evaluatio istrumets are traslated ito the participats primary laguage. From time to time we have ecoutered difficulties with traslatio issues, such as durig the applicatio of the WCA i Mozambique, where the majority of health staff at the peripheral level did ot speak Portuguese. I this case, a WCA facilitator iterpreted for respodets i their local laguage. Due to time ad fudig costraits, we caot carry out iterviews with all participats i a leadership program. Therefore our evaluatios most ofte use a purposive sample. Purposive samplig is a form of oprobability samplig i which respodets are selected accordig to a specific pla or purpose. Purposive samplig is differet from probability samplig, whereby each member of the populatio has a equal chace of beig sampled ad the results ca be geeralized to the sample populatio. Purposive samplig is useful for reachig a targeted sample ad whe samplig for proportioality is ot the mai cocer. The disadvatage of a purposive sample is that it is hard to kow how well the sample represets the populatio. It is importat to qualify the fidigs ad ote whether people left out of the sample might behave differetly from those who were selected. For example, i our evaluatios, teams are usually rated ad selected accordig to criteria such as the quality of their actio plas ad their adherece to the SMART 8 priciples for defiig measurable results. These criteria are iteded to esure, as much as possible, maximum variability i the performace of teams icluded i the sample. K e y q u e s t i o s w e a d d r e s s Our leadership program evaluatios are geerally based o a similar set of key questios. This allows us to sythesize the lessos leared across programs ad evaluatios. It is useful to triagulate several data sources for the same questio to verify the iformatio collected. Examples of the types of questios addressed i our evaluatios iclude: 7 While usig service statistics for measurig outcomes is ofte expediet ad ecoomical, there are also challeges from a measuremet perspective. Some of the primary challeges are outlied i Data ad Iformatio Systems: Issues for Performace Measuremet, i Perri et al. 1999, pp. 83 131. 8 SMART: Specific: to avoid differig iterpretatios; Measurable: to allow moitorig ad evaluatio; Appropriate: to the problem, goals, ad strategies of the orgaizatio; Realistic: achievable, challegig, ad meaigful; ad Time boud: with a specific time period for achievig them. M S H O C C A S I O N A L PA P E R N O. 8 1 3

What techical assistace approaches ad tools were used i deliverig the LDP? What other results were achieved that are urelated to addressig their challeges? What orgaizatioal challeges did the teams address through this program? What processes did participats establish to address their challeges (for example, team meetigs, aalyzig local data, cosultig iteral stakeholders)? Did teams develop actio plas to address their challeges? If so, were all plaed activities carried out? Were activities implemeted that were ot icluded i the actio pla? How did the teams ad their orgaizatios moitor their progress i addressig their challeges? To what extet did the teams ad their orgaizatios achieve their expected results? What motivated participats to achieve their desired results? What preveted them from achievig them? Did the teams cotiue to work together to address aother challege after the leadership program eded? If so, what processes were used ad how were they similar to or differet from how they worked together durig the LDP? What motivated their participatio ad commitmet after the formal program eded? To what extet was cotet from the LDP shared with staff members who did ot participate i it? I what ways did the LDP affect the performace of idividuals, teams, ad the orgaizatios as a whole? M S H O C C A S I O N A L PA P E R N O. 8 1 4

S E C T I O N I I I : Lessos Leared from a Evaluatio Perspective We leared three kids of lessos i the course of this work: lessos about measurig logterm outcomes, lessos about measurig work climate, ad lessos about evaluatig the VLDP. W h at w e l e a r e d f r o m m e a s u r i g l o g - t e r m o u t c o m e s Lesso 1: The quality of participats M&E systems is a critical factor i the ability to show ad sustai results at the service delivery level. The cliet orgaizatio s M&E system determies its ability to collect ad process data ad provide accurate iformatio. The ability to track chages depeds o the quality ad accuracy of service statistics ad, as i the case of the leadership program i Aswa, Egypt, the appropriateess of the idicators selected. Teams i the Egypt program that had selected family plaig as their challege used a idicator madated by the Miistry of Health (MOH), which did ot follow iteratioally accepted guidelies for calculatig couple years of protectio. Moreover, the idicator they used did ot accurately measure the teams family plaig service results, i some cases uderreportig results, i other cases overreportig them. Lesso 2: Workig with cliets to moitor progress ad measure fial results as a itrisic part of a leadership program ca create a culture of iformatio that outlasts the program itself. The leadership program carried out by the MOH ad the M&L Program i Nampula Provice, Mozambique, from 2004 to 2005 demostrates this priciple. The program used simple maagemet ad leadership tools to assist six districts ad 11 health uits ad their commuities to improve discrete aspects of service delivery. MOH ad LDP facilitators supported the health uits i developig actio plas with measurable idicators ad feasible moitorig mechaisms. I the process the program created a culture of iformatio amog health care maagers ad providers i a settig where performace had ever bee moitored ad M&E skills were very limited. This culture of iformatio represeted a ew way of operatig at the peripheral level based o the use of data for plaig, verificatio, ad reflectio. The fial program evaluatio i 2005 showed that, for the most part, the health uits achieved surprisig results, give the low level of quality i the uits ad their extreme lack of resources. For example, health services improved i 10 of 11 health uits, resultig i shorter waitig times for patiets, improved clealiess ad hygiee, icreased biosecurity, improved quality of food, costructio of a materity waitig home, ad more accurate medical records. Furthermore, the health uits were successful i measurig their performace usig tools they had developed themselves. Table 4 shows the types of results that were moitored ad reported by health uits i this program. Prior to the program, teams from the participatig health uits had miimal skills i developig idicators ad moitorig performace. They leared to develop idicators that were appropriate to their challeges yet feasible to measure i their settig. For example, they had to fid a way to defie the criteria for clealiess, hygiee, ad biosecurity i a settig where they had few or o cleaig supplies ad the health uits were i a state of decay. While developig these idicators was difficult ad time-cosumig, participats were much more likely to take owership of the process because they had helped defie how their success would be measured. M S H O C C A S I O N A L PA P E R N O. 8 1 5

Table 4. Types of Leadership Results Documeted by Selected Health Uits i Mozambique Name Health Uit Type* Challeges Reported Results, Nov. 2003 Oct. 2004 Mecota Health Ceter Health Ceter Type 1 Icrease percetage of births i the health facility from 25% to 35% Icrease availability of water from 2 to 12 hours/day Icreased percetage of births i the health facility from 25% to 35% Completed costructio of materity waitig home i Nov. 2004 Dug but did ot complete well Namialo Health Ceter Health Ceter Type 1 Icrease average mothly percetage of cotraceptive users from 15% to 50% of quota defied by the Provicial Directorate of Health (774 users) Icreased average mothly percetage of cotraceptive users from 15% to 38% i first 9 moths of 2004 Ilha de Moçambique Health Ceter Health Ceter Type 1 Improve biosecurity Improve clealiess ad hygiee Icreased percetage of biosecurity criteria met from 16% to 83% Icreased percetage of clealiess ad hygiee criteria met from 11% to 67% Lumbo Health Ceter Health Ceter Type 2 Improve iteral ad exteral clealiess ad hygiee Improve the diet of ipatiets Icreased percetage of clealiess ad hygiee criteria met from 20% to 74% Costructed biomedical waste cotaier Paited the health ceter Built a kitche Nacala Porto Geeral Hospital Regioal Hospital Icrease umber of births i the health facility Improve quality of ipatiet care Icreased mothly average umber of births i the facility from 302 for first 6 moths of 2003 to 350 for first 6 moths of 2004 Data ot reported o ipatiet care Carapira Health Ceter Health Ceter Type 1 Reduce waitig time for pediatric visits Reduce umber of errors i ipatiet registry ad Kardex (medicatio registry) Reduced waitig time for pediatric visits by 2.5 hours Reduced errors i the ipatiet registry from 9 i Dec. 2003 to 0 i Oct. 2004 Reduced errors i the Kardex from 100 i Dec. 2003 to 0 i Oct. 2004 * A Type 1 Health Ceter offers basic primary health services ad provides ipatiet materity care for vagial deliveries; it may have a medical techicia o staff but does ot have a physicia. A Type 2 Health Ceter does ot offer ipatiet materity services ad has o techical medical staff. A regioal hospital has physicias ad offers a rage of medical ad surgical services. M S H O C C A S I O N A L PA P E R N O. 8 1 6

A urse at the Ilha de Moçambique Health Ceter explaied how the ew culture of results is cotributig to better services: We did t ecourage our staff to step back from their work ad evaluate their activities before this program. We had plety of ideas, but we asked for help ad moey whe we ecoutered problems ad we ever evaluated ourselves. Now the hospital wards have a healthy spirit of competitio regardig which ward has the best results whe their activities are matched agaist the criteria that we all established. M&L coducted a evaluatio five moths after the leadership program eded. The evaluatio revealed that several teams were still usig their tools to moitor progress ad, importatly, they were able to chart their post-program performace compared to performace durig the program. Lesso 3: Not all leadership results are quatifiable. A mixed method approach is the most appropriate way to measure the results of a leadership program. Because leadership developmet is as much about the process as it is about the results, evaluators should always cosider usig a balaced mix of qualitative ad quatitative methods to collect data. Evaluatios that rely solely o quatitative data such as service delivery results may coceal importat chages i team dyamics ad their iteractios with the larger system aroud them. The PLP, LEAG, ad SLI have bee i the forefrot of developig ways to evaluate leadership developmet i the health sector. They have promoted the importace of triagulatig methods ad worked to develop a coceptual guide for assessig leadership developmet. The result is the EvaluLEAD framework, which outlies the evocative ad evidetial lies of iquiry ecessary to capture outcomes at the idividual, orgaizatioal, ad societal levels (Grove 2002 ad 2005). Our use of mixed methods was ispired by this framework. The Mozambique evaluatio provides a good example of a balaced mix of methods ad data sources, icludig: reports ad other documets related to the Health Systems Stregtheig (HSS) Program; reports from the November 2004 evaluatio workshop i Nampula Provice o health uits progress i implemetig their actio plas; reports o the progress of the MOH s Aual Operatioal Plas for 2004 ad the first trimester of 2005; observatios durig site visits i November 2004 to Carapira, Lumbo, ad Mecota health uits ad follow-up visits i April 2005 to Carapira, Lumbo, Mecota Mossuril, ad Ilha de Moçambique health uits; results of the WCAs at baselie ad after completio of the program i the Directorate of Admiistratio, Directorate of Reproductive Health, ad participatig health uits; commets from program participats i the LeaderNet 9 virtual forum Achievig Results i Low-Resource Settigs: Stories from Mozambique held i Jue 2005; 9 LeaderNet is a global commuity of practice a group of practitioers who regularly egage i sharig ad learig i a specific techical area that was developed durig the M&L Program ad cotiues to be implemeted i LMS. It operates usig both face-to-face meetigs ad through the Web, telephoe, ad fax, depedig o the eeds of its members. LeaderNet serves as a resource for maagers ad other professioals seekig to improve their leadership ad maagemet skills. Participats i all LMS programs, LDPs, VLDPs, ad semiars offered by the Global Exchage Network (GEN) for Reproductive Health, amog other programs, are ivited to joi LeaderNet: http://www1.msh.org/projects/lsm/programsadtools/ LeadigAdMaagig/LeaderNet.cfm. M S H O C C A S I O N A L PA P E R N O. 8 1 7

semistructured iterviews with: selected participats i the Challeges ad Quality Programs i Nampula Provice; staff i the health uits who did ot participate i the workshops; facilitators of the Challeges ad Quality Programs; selected iformats from M&L Mozambique staff ad key maagers i the MOH; represetatives of the iteratioal door commo fud (SWAP). Aother example comes from the pilot LDP i Aswa, Egypt, carried out from 2002 to 2003. This program was delivered to 10 teams of doctors ad urses at the district ad health facility levels. Expected outcomes were defied as improved climate ad improved family plaig, preatal, ad postpartum care services. The evaluatio i 2003 relied etirely o service statistics to measure outcomes. Although sigificat improvemets were measured i may of the outcome idicators related to health services, the evaluator was ot able to explai i ay detail what had led to these chages. I 2004, a year after the program i Egypt eded, MSH evaluated the replicatio of the same program with aother set of district ad health facility teams. This time the program was implemeted by the Aswa Goverorate itself with o fiacial support from M&L ad with limited, log-distace techical assistace. For this evaluatio, a mix of qualitative ad quatitative methods was used to capture chages i outcomes. Quatitative methods were agai based o a aalysis of service statistics. Qualitative methods icluded semistructured iterviews ad focus group discussios with participats ad facilitators. These iterviews provided isight ito the processes that took place withi the district ad facility teams ad throughout the health system durig the replicatio of the leadership program. This secod evaluatio provided a much richer accout of program results, with evidece of chages i services coupled with isights to help explai how the trasformatio i service delivery was associated with the leadership developmet process. Lesso 4: Carefully selectig iformats ad creatig a safe eviromet are essetial to producig high-quality data. Whe selectig key iformats or focus group members, it is importat to keep a few poiters i mid. First, respodets should represet the characteristics of the larger team that participated i the program. For example, if the team icluded doctors, urses, couselors, ad outreach workers, the the sample of respodets should iclude these types of persoel. Secod, to promote objectivity, the program facilitators or program maager should ot choose the respodets. They should be selected istead by the evaluator accordig to criteria established i the evaluatio protocol. I additio, coductig focus group discussios with participats ad oparticipats is useful to gauge the idirect effects o leadership capacity i the two groups ad cofirm the accuracy of the resposes from those who participated i the program report. There are some risks associated with coductig focus group discussios ad iterviews at the participats workplaces. If a respodet discusses work relatioships with his or her peers or supervisors, this iformatio must be kept cofidetial. For this reaso, evaluators should choose iterview questios with care ad make sure the settig is private. Focus groups should be homogeeous, or it may be difficult to elicit cadid resposes. For example, staff ad subordiates should be grouped together i oe focus M S H O C C A S I O N A L PA P E R N O. 8 1 8

group or set of focus groups, ad maagers, directors, ad supervisors should be icluded i a separate focus group. Fially, the evaluator eeds to establish a cofidetial atmosphere where there is o risk of retributio to ecourage ope discussio ad hoest iput. While focus groups ad iterviews may be recorded, the verbatim trascripts are oly for the evaluator s use ad should ot be shared with respodets peers or supervisors. Lesso 5: Teams ofte eed assistace to develop high-quality actio plas that ca serve as a evaluatio tool. Durig the leadership program, each team selects a challege such as improvig delivery of family plaig services, ad develops a actio pla to idetify the activities eeded to achieve the desired results. The team also defies idicators for measurig baselie ad fial results. Thus the actio pla liks program iputs with desired outcomes. I theory, this process provides a soud basis for measurig outcomes. I practice, however, it ca be problematic to deped o teams to supply data related to their outcomes. Sometimes the data are ot available whe it is time to evaluate progress. Except i programs that use service statistics for moitorig results, teams may ot moitor their performace usig the idicators i their actio plas. There are several reasos for this problem. I some cases, team members chage after the program eds. This is especially commo i the public sector. May teams do carry out their plas but may moitor performace usig orgaizatioal-level idicators rather tha the idicators i their actio plas. I other cases, the team does ot itegrate its actio pla ito the orgaizatio s aual pla, so there may be iadequate resources to moitor usig actio pla idicators. Sometimes, key stakeholders i the orgaizatio are ot supportive, so teams either do ot fully carry out their actio plas or are ot motivated to moitor their performace. Fially, while some teams produce actio plas that are writte logically ad clearly with well-distributed resposibilities, well-defied idicators, SMART expected results, ad appropriate data sources i other cases, it is uclear how activities i the pla will lead to achievig the desired results. A well-developed actio pla is more likely to be implemeted ad its idicators used tha a poorly defied pla. Our prerequisite for usig a actio pla as the basis for a evaluatio is esurig that the pla meets the SMART criteria. To achieve this, the leadership programs have egaged M&E Uit staff to review ad provide feedback to participats o their actio plas, a feature of the VLDP that is yieldig icreasigly better results. The purpose of the review is to check that (1) the selected challege is immediately actioable ad will cotribute to the iteded results, (2) the activities logically address the challege, ad (3) the proposed idicators are appropriate to measure progress. To make sure the pla meets the SMART criteria, teams are give guidace usig the followig questios: Are the missio ad visio of the team clear, especially as they relate to the orgaizatio s overall missio? Is the challege selected by the team clear? Are the desired results defied by the team clear ad achievable withi six to ie moths? Are activities logically related to the desired results? Are measurable idicators defied? Is a time lie or time frame for implemetatio idicated? M S H O C C A S I O N A L PA P E R N O. 8 1 9

Are huma ad fiacial resources idicated i the pla ad are they available? We have also developed materials o M&E for facilitators to use durig the leadership program to support those resposible for providig M&E guidace to participatig teams. Please see http://erc.msh.org/leaderet/assets/ files/m_e_guide_to_vldp_ad_ldp_fial_ 2007.doc W h at w e l e a r e d f ro m m e a s u r i g w o r k c l i m at e MSH has used the WCA to measure work climate ad climate chage durig most faceto-face leadership programs ad all VLDPs coducted to date. Based o this experiece, we offer several lessos leared about the use of the tool ad the quality of the data it produces. Lesso 6: Improvemets i climate caot be attributed oly to the effects of the LDP or VLDP due to iterveig variables ad the log time frame of the LDP. Experiece has show that as teams lear to use leadership ad maagemet practices to address their challeges, work climate teds to improve. This is true for teams from NGOs ad the cetral, provicial, ad local levels of the public health sector. Nevertheless, there are limits to demostratig chages i climate ad attributig them to the itervetio. The log time frame of the LDP (up to a year) meas that other evets, such as political or iteral orgaizatioal chages, ca impact climate for better or worse, despite the effects of the LDP. I the ideal, our evaluatios would use a cotrol group; however, we caot radomly assig participats to such a group. Participatio i a leadership program is ot idiscrimiate ad teams are ot uiformed; teams that participate are either self-selected or are recommeded by a orgaizatioal executive or admiistrator or a door. Lesso 7: For programmatic ad evaluatio purposes, it is ideal to recruit itact teams for the LDP ad VLDP. The LDP tries to recruit itact teams. A itact team is oe whose members have worked together before begiig the LDP ad will cotiue to work together after the program eds (although all members of the team do ot have to be i the same locatio). Itact teams are more likely to carry out their actio plas, ad the behaviors they adopt durig the program ted to be sustaied withi the group afterwards. They are also more resposive to the WCA survey ad easier to follow after the program eds. Teams that are formed artificially for the purpose of participatig i a LDP (for example, a group of supervisors from various levels of a public- or private-sector orgaizatio) have more difficulty i implemetig their plas ad measurig their progress after the program eds. These teams are ofte made up of idividuals from geographically or admiistratively dispersed groups. They may work well together durig the program, but whe they retur to their ormal routies after its completio, they may o loger be motivated to work together as a team because they are ot, i fact, a team. Teams that disitegrate after the program eds are usually lost to follow-up, which egatively affects our ability to measure leadership outcomes. All members of a work group must fill out the WCA for evaluators to obtai a valid measure of work group climate. Ad because itact teams have a history of workig together, they ca respod to the WCA items i a meaigful way. Thus, ot oly for effective program desig but also for evaluatio purposes, it is preferable to recruit itact teams for leadership programs. Despite the beefits of erollig itact teams, ew kids of teams have emerged i the LDP ad VLDP i respose to the eed to stregthe M S H O C C A S I O N A L PA P E R N O. 8 2 0

multisectoral coordiatio. Examples iclude atioal- or district-level HIV & AIDS teams, teams workig regioally to stregthe huma resource maagemet, teams ewly formed by decetralizatio, ad laboratory maagemet teams. If oitact teams participate, the baselie WCA should be admiistered oly after the teams have worked together for three or more weeks. For example, i workig with ew laboratory maagemet teams i a program i Ugada, the LDP facilitators admiistered the WCA durig the secod LDP workshop istead of the first. I the VLDP the fixed structure of the olie course makes such flexibility difficult. At the same time, the emphasis o usig the baselie results of the WCA as a itervetio i ad of itself ca be useful for ew teams as they lear to work together. Lesso 8: Because the WCA is applied by program facilitators, the quality of the work climate data depeds o their ability to explai to participats the purpose of the tool ad how it is used. Usig work climate as a itermediate outcome of a leadership program ca be challegig. Participats must uderstad how positive climate is created ad that improved climate is a expected outcome liked to participatio i the program. The purpose of usig the WCA must be properly explaied by program facilitators; otherwise respodets may ot uderstad the value of the tool. Without this uderstadig, some participats may perceive climate as a measure that serves for reportig to a door, ot as a essetial igrediet i developig a actio pla ad improvig team performace. Moreover, if participats are ot motivated to complete the WCA survey, the program evaluatio will likely suffer from poor respose rates. For example, durig the replicatio of the LDP i Egypt, whe Egyptia facilitators delivered the program to a ew set of district ad health facility teams, they did ot use the WCA because they thought it was ot essetial to the program. As a result, the fial evaluatio lacked climate data to use for compariso purposes, although improved climate was oe of the iteded outcomes of the program. To make the results of the WCA meaigful to teams, we have leared to weave the baselie results ito the cotet of the leadership program. Facilitators help teams iterpret their climate data ad develop strategies to improve their climate usig leadership practices. Recetly, the use of the WCA was greatly improved durig a VLDP i Peru for staff from a muicipality-based program, aother USAID cooperatig agecy, ad a local NGO collaboratig with the program. The VLDP facilitator icluded a review of the WCA tool ad aalysis of the baselie climate results i oe of the early modules ad ecouraged participats to discuss the tool, the importace of climate, ad the actios eeded to improve it. That VLDP also icluded pre- ad post-course WCA surveys as a prerequisite for receivig the course diploma. Both strategies helped icrease respose rates substatially. Fially, it is importat that participats uderstad the five-poit scale used i the WCA so they ca respod correctly. This scale has occasioally caused cofusio because participats are ot familiar with scales or are ot used to ratig their experieces. Facilitators must carefully explai the scale ad the meaig of the values. After the WCA, facilitators should help participats iterpret their results ad determie ways to improve their work climate. Fially, although MSH staff validated the WCA i differet coutries ad laguages, populatios with low literacy ad those ot accustomed to self-admiistered questioaires ted to have trouble respodig. Some participats M S H O C C A S I O N A L PA P E R N O. 8 2 1

may eed extra attetio durig the applicatio of the tool. For example, the leadership program i Mozambique relied o a local facilitator to apply the Portuguese versio of the WCA amog MOH persoel. This facilitator offered simultaeous traslatio ito the local laguage for staff that were oliterate or spoke a laguage other tha Portuguese. W h at w e l e a r e d f ro m e va l u at i g t h e V L D P M&L coducted follow-up evaluatios of seve VLDP programs (three i Lati America, two i Africa, oe i the Caribbea, ad oe i Brazil) to documet the mai outputs ad outcomes produced by the participatig teams. The evaluatios focused o two elemets: results achieved through the implemetatio of actio plas ad chages i work climate. Methods icluded a review of all actio plas developed durig the program ad, where possible, a assessmet of idicator data from the plas. I additio, we solicited resposes to a e-mail questioaire from the whole team as well as a represetative from each team, ad coducted telephoe iterviews with a member of selected teams. We also aalyzed data from the applicatio of WCA that were posted o the VLDP website by teams. Several lessos emerged from this experiece about the most effective methods for obtaiig iformatio from virtual teams. Lesso 9: VLDP follow-up evaluatios should take place six to ie moths after program completio. The timig of the follow-up evaluatio is crucial for capturig high-quality iformatio from participats. The more time that passes after completio of the VLDP, the lower the respose rate to e-mail questioaires ad iterview requests ad the greater the risk of recall bias. At the same time, teams eed eough time to practice ew skills ad implemet their actio plas before the follow-up evaluatio takes place. The evaluator must select a time frame that balaces recall ad results. I our experiece, six to ie moths after program completio is usually sufficiet time for most teams to implemet their plas. Lesso 10: VLDP e-mail questioaires should be set to a poit perso from the team, followed by a telephoe iterview with carefully chose iformats. The e-mail questioaire solicits iformatio from VLDP teams o measurable progress o the actio pla (icludig idicator data) ad processes the team used to implemet its pla. Without adequate advace preparatio by iformats, however, the e-mail questioaire is a poor mechaism for gatherig process ad outcome data. Our biggest challege has bee low respose rates. We have tried differet approaches to overcome this problem, with varyig degrees of success. We have tried sedig the questioaire to the team leader aloe; however, we foud that this perso ofte lacks iformatio o idicator data. The iformatio is much richer whe all team members fill out the questioaire, but respose rates are usually poor whe the questioaire is set to the etire team. The best approach we have foud is to select a poit perso who will seek ad compile resposes from all team members. Teams eed to be iformed i the early stages of implemetatio of the program that e-mail questioaires will be set after the completio of the program. They should select a poit perso to dissemiate the questioaire ad collect resposes. But while usig a poit perso has improved respose rates, there is o guaratee that the questioaire will actually be completed by all members of the team. The ext step is to coduct a i-depth telephoe iterview with oe team member from each of several teams. The purpose of the iterview is to verify the iformatio from the M S H O C C A S I O N A L PA P E R N O. 8 2 2

questioaire ad probe further about progress i results ad work climate. The best strategy for selectig teams for telephoe iterviews is to categorize them ito high, medium, ad low performers accordig to criteria such as the quality of the actio pla. The two to three teams from each category are selected for the telephoe iterviews, ofte usig geographic criteria. This process allows us to capture variatios i the cohort ad compare resposes across performace levels. Sice the respodet s familiarity with the team s progress is essetial for a high-quality telephoe iterview, each VLDP team should select a spokesperso who will be prepared to respod to telephoe iterviews or requests for iformatio throughout the moitorig ad follow-up evaluatio period. The spokesperso should be elisted early i the program ad made aware of his or her resposibilities to represet the team. M S H O C C A S I O N A L PA P E R N O. 8 2 3

S E C T I O N I V : Recommedatios about Measurig Leadership Effective leadership developmet is reflected by improved work climate, stregtheed systems, ad better performace i the delivery of health services. I additio, chages i behaviors, values, ad orms take place i participatig teams ad their orgaizatios, which are harder to make explicit. Our job as evaluators is to measure both the results achieved ad explore the process used to reach them i order to develop well-fouded isights ito the likely causes of chage ad potetial alterative explaatios. We have leared that to do this well evaluators should keep several poits i mid. Desig the leadership program with the evaluatio i mid. Evaluatios are highly depedet o the way i which the program facilitator itroduces ad uses measuremet tools that supply data for the evaluatio. For example, program facilitators who icorporate the WCA ito the program i a way that is meaigful to participats ca icrease their motivatio to respod to the questioaire ad use the results to moitor their progress i workig together. Participats eed to be motivated to use the idicators i their actio plas for moitorig performace ad measurig chage i relatio to a baselie. Moitorig results eeds to become a program strategy so that participats lear to value the collectio ad use of data to measure their progress. All leadership programs should use simple measuremet istrumets that participats will fid useful i their work. This priciple is especially importat if the program relies o participat data to measure its successes or failures. As we have oted, relyig o the cliet orgaizatio s M&E system ca either ehace or weake the ability to measure leadership outcomes. Participats from orgaizatios with weak M&E systems are ofte uable to provide a suitable baselie measure. Or they may ot use the idicators i their actio plas to moitor performace, so post-program data are uavailable to measure chage i relatio to a baselie. This is particularly the case whe the team has idetified as its priority challege a orgaizatioal process or system, which is usually assessed usig qualitative methods. It happes less ofte whe the team has idetified a service delivery related challege, which ca be assessed usig available service data. The value of qualitative iformatio has led us to stregthe our qualitative approaches ad cosider offerig techical assistace i M&E to cliet orgaizatios ad participatig teams. Balace the eeds of doors without sacrificig the cotet of the evaluatio. As program desigers ad evaluators, we must balace the eed to report short-term results to doors ad the eed to oriet participats to select sigificat targets that will cotribute to their desired outcomes ad ca be measured after the program has eded. I geeral our leadership programs have bee fuded o a aual basis. This limitatio has iflueced data collectio methods ad the scope ad timig of evaluatios. Limited time frames ofte ihibit our ability to measure outcomes i terms of broad orgaizatioal chage. Yet there is pressure from doors to report this kid of result durig a short project. Furthermore, doors ted to prefer quatifiable evidece of outcomes eve though qualitative methods are ofte more appropriate ways to capture chage. Programs ca respod to these pressures i several ways. Programs should help participats develop performace bechmarks that ca be M S H O C C A S I O N A L PA P E R N O. 8 2 4

measured as the program is carried out, especially if outcomes are likely to chage slowly ad if it will be difficult to gather process data from participats after the program has eded. Alteratively, the immediate leadership outputs (leadig practices) ad itermediate outcomes (work climate ad maagemet systems) described above ca be measured. Either approach may satisfy the eed to report short-term data without udermiig the teams ambitio to reach loger-term outcomes. Fially, more systematic reportig of results usig mixed methods may meet doors eed for hard data, while highlightig the trasformatio process that occurred amog participats ad teams. Make use of ope systems theory. Causality is complex, especially whe a result depeds ot oly o improved leadership but also o other maagemet capacity such as the availability of commodities or quality of supervisio. There are multiple exteral iflueces o participats i a leadership program ad results at each level (e.g., output, outcome) ca be cocurret. This meas that coceptual models that assume a liear relatioship betwee iputs, processes, outputs ad outcomes have a limited applicatio. Although we have developed models to orgaize ad pla our project desigs ad evaluatio strategies, the fit of these models with leadership developmet is sometimes imperfect. Takig a ope systems approach may be the aswer for the desig of future MSH leadership projects ad evaluatios. We recogize that leadership developmet is a ope system cosistig of a complex iterplay amog activities, idividuals, teams, orgaizatios, ad commuities. Our evaluatios are already desiged to take ito accout both the expected ad uexpected results of this process. Ad our aim is ot to attribute causality of results solely to program iputs but rather to measure cotributios to results. I the future, we will cosider usig well-tested frameworks that take a ope systems perspective, such as EvaluLEAD, as structures to guide the capture of qualitative iformatio about ad isights ito the effects of leadership programs. Fully explore the leadership process to better uderstad the outcomes. The differece betwee leadership developmet ad other approaches to improvig health services is i the process used. But to truly uderstad the leadership process, we must capture iformatio about how people thik, feel, ad act. For example, we have leared the importace of capturig iformatio o what we call the collaborative improvemet effect. Through the collective aalysis of their challege ad the selectio ad implemetatio of itervetios to address it, team members lear from each other, develop ew values ad habits, ad experiece setbacks as well as progress i their selected health area. This collaborative improvemet uderscores the value of trackig itermediate outcomes ad leadership outputs ad explais why collectig participats perceptios of the leadership process is vital to explaiig outcomes. To evaluate a leadership program without capturig the voices ad reflectios of participats would be missig a key piece of the picture. M S H O C C A S I O N A L PA P E R N O. 8 2 5

Coclusio Developig maagers who lead i the health sector provides a fresh perspective o measurig performace improvemet. Oe implicatio is that focusig more o the leadership process meas producig larger amouts of qualitative iformatio. I terms of aalysis, it could be useful to determie whether the themes idetified i the evaluatio are reflected i existig models such as Kotter s Eight Step Model, which is a importat uderpiig of the cotet of the LDP (Kotter 1996, Kotter ad Cohe 2002). A approach such as a fitgap aalysis could help frame the issues, guide the subsequet aalysis, ad better explai the evaluatio fidigs. 10 I our leadership programs, performace improvemet occurs at two levels: itermediate chages withi the team (work climate) ad withi the orgaizatio (systems) ad outcomes at the service delivery level. The teams determie their mai challege ad agree o their expected outcomes. They choose what is feasible to achieve ad yet importat eough to make a differece i the delivery of services. The they set out o a jourey that will revolutioize how they see each other as professioals, how they work together as colleagues, ad how much they ca achieve together. This paper describes how to capture the richess of the process ad uearth some of the tagible accomplishmets that result i order to iform our job as evaluators. 10 Fit-gap aalysis is a tool for carryig out a cotet aalysis. This approach idetifies parallels ad discrepacies betwee the model of iterest ad the themes ucovered i the evaluatio. Itersectios betwee the model ad evaluatio themes ca be mapped i a diagram or table to elucidate the fidigs. M S H O C C A S I O N A L PA P E R N O. 8 2 6

Resources Ceraso, Mario, et al. Evaluatig Public Health Leadership Traiig i Wiscosi: Beyod Participat Report. Abstract of a presetatio (review of the literature o public health leadership traiig) at the America Public Health Associatio aual meetig, Nov. 2006. http://apha.cofex.com/apha/ 134am/techprogram/paper_136197.htm Golema, Daiel. Leadership That Gets Results. Harvard Busiess Review March April 2000, pp. 78 90. Grove, J., B. Kibel, ad T. Haas. EvaluLEAD: A Guide for Shapig ad Evaluatig Leadership Developmet Programs. Oaklad, CA: Sustaiable Leadership Iitiative, Public Health Istitute, 2005. http://www. phi.org/pdf-library/evalulead.pdf Grove, J., ad LEAG. The EvaluLEAD Framework Examiig Success ad Meaig: A Framework for Evaluatig Leadership Developmet Itervetios i Global Health. Oaklad, CA: Populatio Leadership Program, Public Health Istitute, 2002. Haum, Kelly M., Jeifer W. Martieau, ad Claire Reielt, eds. The Hadbook of Leadership Developmet Evaluatio, Sa Fracisco, CA: Jossey-Bass, 2007. Heifetz, Roald A., ad Doald L. Laurie. The Work of Leadership. Harvard Busiess Review Ja. Feb. 1997, pp. 123 34. Kotter, Joh P. Leadig Chage. Bosto, MA: Harvard Busiess School Press, 1996. Kotter, Joh P., ad Da S. Cohe. The Heart of Chage: Real-Life Stories of How People Chage Their Orgaizatios. Bosto, MA: Harvard Busiess School Press, 2002. Laschiger, Heather K. Spece, Joa Fiega, ad Judith Shamia. The Impact of Workplace Empowermet, Orgaizatioal Trust o Staff Nurses Work Satisfactio ad Orgaizatioal Commitmet. Health Care Maagemet Review 2001, Vol. 26, pp. 7 23. LeMay, Nacy, Nacy Vollmer, ad Aliso Ellis. Evaluatig Leadership Developmet ad Orgaizatioal Performace. I The Hadbook of Leadership Developmet Evaluatio, eds. Kelly M. Haum, Jeifer W. Martieau, ad Claire Reielt. Sa Fracisco, CA: Jossey-Bass, 2007, pp. 228 60. Litwi, George H., ad Robert A. Striger, Jr. Motivatio ad Orgaizatioal Climate. Bosto, MA: Harvard Uiversity Press, 1968. Luoma, Marc, ad Laure Voltero. Performace Improvemet: Stages, Steps ad Tools. Chapel Hill, NC: ItraHealth, 2002. http://www.prime2.org/sst/ Maagemet Scieces for Health. Creatig a Work Climate That Motivates Staff ad Improves Performace. The Maager 2002, Vol. 11, No. 3, pp. 1 22. http://erc.msh.org/maipage.cfm?file=2.1.55. htm&module=leadership&laguage=eglish Maagemet Scieces for Health. From Visio to Actio: A Course o Moitorig ad Evaluatio for Facilitators of Leadership Developmet Programs. LeaderNet Workshop Resources, March 2007. http://erc.msh. org/leaderet/assets/files/m_e_guide_to_vldp_ ad_ldp_fial_2007.doc Maagemet Scieces for Health. Maagers Who Lead: A Hadbook for Improvig Health Services. Cambridge, MA: Maagemet Scieces for Health, 2005a. Maagemet Scieces for Health. Work Climate Assessmet Tool ad WCA Guide for Facilitators. 2005b. I the Health Maager s Toolkit at: http://erc.msh. org/maipage.cfm?file=96.9htm&module=toolkit& laguage=eglish Perri, Edward B., Jae S. Durch, ad Susa M. Skillma, eds. Health Performace Measuremet i the Public Sector: Priciples ad Policies for Implemetig a Iformatio Network. Washigto, DC: Natioal Academy Press, 1999. Perry, Cary, et al. Validatig a Work Group Climate Assessmet Tool for Improvig the Performace of Public Health Orgaizatios. Huma Resources for Health 2005, Vol. 3, No. 10, pp. 1 8. http://www. huma-resources-health.com/cotet/3/1/10 M S H O C C A S I O N A L PA P E R N O. 8 2 7

Reielt, Claire, et al. Fial Report: Leadership Matters: A Evaluatio of Six Family Plaig ad Reproductive Health Leadership Programs. Developmet Guild/DDI, 2005. http://www.leadershiplearig.org/system/ files/packardgates_fialreport_web_0.pdf Striger, Robert A., Jr. Leadership ad Orgaizatioal Climate. Upper Saddle River, NJ: Pretice Hall, 2002. Umble, Karl, et al. The Natioal Public Health Leadership Istitute: Evaluatio of a Team-Based Approach to Developig Collaborative Public Health Leaders. America Joural of Public Health 2005, Vol. 95, No. 4, pp. 641 44. http:// www.pubmedcetral.ih.gov/articlereder. fcgi?artid=1449235 Uiversity of Southampto. Leadership Evaluatio: A Impact Evaluatio of a Leadership Developmet Programme. Southampto, UK: Uiversity of Southampto Health Care Iovatio Uit ad School of Maagemet, 2004. http://eprits.soto.ac. uk/14067/01/leadership_evaluatio_report.pdf M S H O C C A S I O N A L PA P E R N O. 8 2 8

Ackowledgmets The authors would like to thak the participats i leadership developmet programs aroud the world whose cotributios to evaluatios formed the basis for the experieces ad lessos documeted i this paper. We would also like to thak our MSH colleagues for their visio ad hard work i developig, refiig, ad coductig leadership developmet programs as well as for their commitmet to evaluatio. Thaks also to USAID for support of ad fudig for this Occasioal Paper. We welcome feedback o this paper. Please direct your commets to Nacy LeMay at lemay@msh.org ad Aliso Ellis at aellis@msh. org. All leadership developmet programs ad evaluatios described i this paper were carried out uder the M&L ad LMS programs with fudig from the US Agecy for Iteratioal Developmet (USAID) Office of Populatio ad Reproductive Health, Bureau for Global Health, award umbers HRN-A-00-00-00014-00 ad GPO-A-00-05-00024-00. The opiios expressed herei are those of the authors ad do ot ecessarily reflect the views of USAID. Maagemet Scieces for Health 784 Memorial Drive Cambridge, MA 02139-4613 USA Telephoe: 617.250.9500 Fax: 617.250.9090 Website: www.msh.org M S H O C C A S I O N A L PA P E R N O. 8 2 9