Evaluation of National Family Health Promoter Program Prepared for the Department of Health Promotion Directorate of Community Health Services

Evaluation of National Family Health Promoter Program Prepared for the Department of Health Promotion Directorate of Community Health Services Ministry of Health Republica Democrática de Timor Leste Final Report June 2009 1

This document does not represent the views or opinion of USAID. It may be reproduced if credit is properly given 2

Contents Contents... 3 Executive Summary... 4 Acknowledgements... 6 Introduction... 7 Methodology... 9 Results/Findings:... 11 Section A: Recruitment... 11 Section B: Training... 12 Section C: Implementation... 14 Section D: Supervision and Program Management... 17 Section E: Program Strengths and Limitations, as perceived by different stakeholders... 19 Section F: Competency Tests... 21 Additional Insights from Community Members... 23 Limitations... 24 Conclusion... 25 Annexes... 27 Annex 1. Timeline of PSF Program... 27 Annex 2. Village Management Committee for PSF Program... 28 Annex 3. Members of Field Team... 29 Annex 4. MTT/DTT FGD Participant List... 30 Annex 5. List of Interviewees at National Level... 31 Annex 6. Interview Guide and Competency Test for PSFs... 32 Annex 7. Interview Guide and Competency Test for Health staff... 35 Annex 8. Interview Guide for Community Leaders... 38 Annex 9. Interview Guide and Competency Test for mothers... 40 Annex 10. FDG Guide for MTT/DTTs... 42 Annex 11. Interview Guide for national level... 44 3

Executive Summary The Promotor Saude Familia (or, in English, Family Health Promoter) program is a key intervention of the Ministry of Health (MOH), Government of Timor Leste to extend the reach of basic primary healthcare services into community and household level. Managed by the Department of Health Promotion, the program was initially designed for a pilot test in 4 districts over two years (including two in this study). However, SISCa ( Serviso Integrado da Saude Communitaria (or in English Integrated Community Health Care), the initiative of the newly appointed Minister of Health in 2007, required the Promotor Saude Familia (PSF) program to expand rapidly to cover all 13 districts in the country during 2008. In late 2008, the MoH requested that TAIS organize a formal evaluation of the PSF program in preparation for a national level review of health promotion and the upcoming planning process, and submit findings by the end of February 2009. Considering limitations of time and resources, TAIS proposed a primarily qualitative evaluation study, with the participation of many agencies involved in supporting the MOH community-based services initiatives. The proposal was accepted and the evaluation was conducted during January and February 2009. This report documents the objectives, methods and findings of the evaluation, and provides recommendations. The general objectives of the evaluation were to: 1. Understand how well the PSF program is working, compared to expectations expressed in the PSF and SISCa program guidelines and by the MoH 2. Show which aspects are not working well and why 3. Provide recommendations for making the program more effective in achieving health goals. Six sub districts in three districts (Bobonaro, Aileu and Viqueque) were sampled on the basis of partner support and whether the PSF were trained during 2008. Sucos and aldeias were selected based on perceived accessibility. Data was gathered through individual interviews and focus group discussions with different stakeholders, and brief competency tests were administered to PSF, health workers and their trainers. Interviews were conducted with community leaders, mothers, health staff, and PSFs. Focus group discussions were conducted with national and district trainers. Additional interviews were conducted with relevant departments in the MOH national level. Results are organized by the major investigatory themes: recruitment, training, implementation, and management. The main results and recommendations are presented in the table below: Strengths Weaknesses Recruitment Training Implementation Management MOH able to PSFs active at recruitment many monthly SISCa PSFs activities Majority of PSFs selected by health staff MOH able to training PSFs following the 6- day training schedule Training has been popular among PSF Many PSFs have forgotten what topics they were training in PSFs do not work outside of SISCa SISCa provides solid structure to manage PSFs No supervision tools Health staff unclear about 4

Recommendations Recruitment Training Implementation Management their management roles Management committees have never functioned consistently at any level Need regular communication between health staff and community leaders Replacement policy needs to be articulated; community involvement in PSF recruitment needs better definition Use supportive supervision tools to understand and fix areas of PSF weakness Classify PSF by whether they have been given the new training, and ensure one round of new training to all. Start regular internal communication between program staff and PSFs Provide clarity on job description of PSF particularly on what they are expected to do beyond SISCa. Implement supervision tools Conduct monitoring visits Clarify roles Create transparent system for PSF payments Strengthen PSF database to maintain updated name-based list of recruited and trained PSF at sub-district levels; Health workers need training in supervision and support functions. 5

Acknowledgements TAIS and the MOH would like to thank the twelve partner organizations for their involvement in the evaluation in the areas of fieldwork and presentation of preliminary findings to the MOH: Health Alliance International (HAI), Services for the Health in the Asia Africa Region (SHARE), Oxfam, CARE, Cruz Vermelha Timor-Leste (CVTL), Alola Foundation, Alliance of Friends for Medical care in East Timor (AFMET), Medicos Da Mundo, and Concern. 6

Introduction The MOH requested that TAIS organize an assessment of the PSF program which would be ready for presentation prior to a national level review of the Health Promotion Department s strategic annual plan. Specific areas of interest for the MOH centered around four main areas of recruitment, training, implementation and management. The evaluation was expected to examine the following: 1. Training modules - how appropriate they are for the purpose of training PSF, given the PSF program objectives. This was not accomplished in detail because appropriate individuals could not be found for the task. 2. Supervision and management (how PSF are supported, supervised and managed, as perceived by PSF and health staff) 3. Strengths and limitations of the PSF program (accessibility, competence, motivation, achievements, usefulness/value, as perceived by different stakeholders) 4. Awareness of roles of PSF in the community among PSF and among MoH staff at different levels (what PSF are expected to contribute by technical, operational tasks and questions related to reach) 5. Competence in some core aspects of technical/health knowledge/behavior change communication (or interpersonal communication skills) 6. Health knowledge and skills, mainly related to MCHN (PSF compared to community, trainers, health staff, district leadership) 7. Perceptions about what needs to change and how (how PSF program could be strengthened/made more effective) 8. Background and History of the PSF Program PSFs are community health volunteers that are: Suco members that are selected by their community to serve as Family Health Promoters Volunteers-not staff of the Ministry of Health and will not receive monetary awards for their work Registered with the Ministry of Health, participate in MOH approved trainings and demonstrate competency during trainings Motivated to conduct PSF identified health promotion activities (Translated from Tetum, Family Health Promoter Guidelines, MOH Timor-Leste) Broadly guided by the PSF program guidelines of 2005, the program began in 2006 in four pilot districts (Viqueque, Aileu, Manatutu, Liquiça), with an average of three PSF per aldeia (detailed timeline of the program is attached in Annex 1). Three simultaneous and related processes influenced the development of the program in the last year (2008): (1) the production and refinement of training modules for PSFs (2) a cascade training approach involving sets of master trainers and district-level trainers and (3) the beginning of SISCa. As of January 2009, there are over 1,500 PSFs listed a database of trained PSF available with the MOH. When the SISCa program was formally introduced in early 2008, it catalyzed the expansion of the PSF program to the entire country, and had a new standard: 5-7 PSF per suco1. It is this expanded phase of the PSF program that is the primary subject of this evaluation study. 1 Although there seems to be no record for this, this is quoted, and seems justified by the need for five persons at least at the monthly SISCa. 7

Module C, which is the module meant for training the PSF, predates the SISCa initiative, and does not mention the term. While this module does not explicitly list a set of tasks to be performed by the PSF, the training content clearly implies the roles envisaged for the PSF, which are largely encapsulated in the four -dors : Educator, Mediator, Motivator, Provider of Information. What the modules lack are specific operational details: what exactly the PSF is expected to do and how often, spending how much time, to achieve what specific goals, reporting to whom in what way and how often. The modules do not describe how the program will be evaluated. However, the PSF program guidelines state that the M&E sub-committee of the National Program Management Committee (NPMC) should develop tools for M&E before PSF are mobilized. They also state that baseline data should be collected and analyzed and indicators developed, but this did not happen. The intended PSF program management structure consists of management committees from the national level to the suco level that include members from all sectors at each level (Further detail attached in Annex 2). The originally outlined roles and responsibilities of PSFs were as follows (Family Health Promoter Guidelines, MOH Timor-Leste): To assist in the planning, development and coordination of public health activities with the support of health service personnel, community members/health committee and/or PSFP partners To coordinate and implement designated PSF health promotion activities in their communities with groups, families and individuals To inform families and community members about health initiatives and health-related matters whenever the opportunity arises To provide support to prevent health problems To recognize serious illness and refer to local health services To keep simple records of work carried out, collect data and provide information to local health service personnel and relevant PSF partners To work alongside members of the health team/community and organizations to take part in activities such as outreach services, immunization and surveys To treat simple illnesses as per training provided To maintain and ensure re-stocking of IEC materials and supplies To serve as a link between her/his community and health services To serve as a positive role model for their communities by practicing healthy habits To understand the full nature of the PSF work To maintain a PSF code of personal ethics. SISCa was understood as a mechanism to periodically bring health services and health information/ promotion closer to the people. The MoH s goal in 2008 was to have functioning SISCa posts in all 442 villages in the country. To meet this goal, the PSF program had to be scaled up, since PSFs were expected to assist in each monthly SISCa. In early 2008, three regional coordinators two from TAIS and one from HAI, were appointed by the Health Promotion Department to coordinate the implementation of the PSF training and program implementation. They mainly help DPHOs in respective regions coordinate most of the above processes. 8

Since the introduction of SISCa, these responsibilities have expanded to specifically include: Participating in and assisting SISCa Coordinating and organizing SISCa activities Mobilizing communities to attend SISCa. Methodology Given the time constraints a quick qualitative assessment was proposed as a comprehensive evaluation of the PSF program was not possible in the six weeks given to produce findings. The MOH indicated districts with characteristics that they wanted to focus on: The informants identified to be leaders and members of the communities that the PSF serve, the PSF themselves, government health staff, trainers in the PSF program, NGOs supporting the PSF program, and key officials of departments of MoH that either manage or collaborate with the PSF program. Key study questions were discussed and written, then a set of questionnaires developed to interview sampled individuals from each of the above categories (other than NGOs). The questions were simple and open-ended, to facilitate easy asking and responding. Simple competency tests were designed for community members (mothers of under-five children), PSF, health workers and MTT/DTT members. The main comparison groups considered relevant for the study were areas with or without partner support, and PSFs who received or did not receive the new training of 2008. Accordingly, the following districts and sub-districts were selected: District Subdistrict PSF Pilot districts Partner Support Training Viqueque Uatucarbau Yes None None Viqueque Lacluta Yes None Once 2 years ago Bobonaro Maliana No Supported Once 2 years ago Bobonaro Lolotoe No Supported Once 2 years ago Aileu Laulara Yes Supported Twice with revised materials Aileu Liquidoe Yes Supported Twice with revised materials The sampling design was kept simple and the overall sample sizes kept manageable within the available time: Three districts, two sub-districts in each district, one suco and two aldeias (one far and one close to suco) in each sub-district were chosen. At least 10 mothers of under-five children (representing community members), all available PSF (at least three) in each selected suco, xefi suco and two xefi aldeias from each suco and all available sub-district health staff were interviewed. 9

Sampling unit used: Chefe suco / Chefe aldeia Health Post / CHC Staff Focus Group Discussions none none Key Informant Interviews / In-depth Interviews Chefe aldeia of the two sampled aldeia (closed to SISCa, farthest from SISCa), Chefe suco Focal Point for HP Head of CHC Midwife Nurse PSF / CHW none With all available PSFs in suco Community none 10 mothers with children under 5 (5 from closest aldeia, 5 from farthest aldeia) DPHO none One Yes Competency Tests (to all health staff, DTT/MTTs, PSFs, mothers) No Yes Yes Yes DTT / MTT (conducted at national level) 12 DTT 12 MTT Invite DTTs & MTTS from all 13 districts Eight National Level All FGD DTT/MTT participants The final sample consisted of the following: District Subdistrict Suco Mothers PSF Comm Health MTT National Leaders Workers DTTs Viqueque Uatucarbau Afalocai 12 7 3 3 Viqueque Lacluta Uma Tolu 12 13 3 5 Bobonaro Maliana Saburai 10 9 3 3 Bobonaro Lolotoe Guda 10 7 3 4 Aileu Laulara Madabeno 10 8 3 4 Aileu Liquidoe Berelau 10 7 3 4 National- 23 FGD National 8 Total 64 51 18 23 23 8 A team of NGO staff (complete list attached in Annex A) administered all questionnaires and competency tests. These were organized into six teams, one for each sub-district, each of which gathered the main data simultaneously in about five days of field work. After all data collection was completed, data were analyzed semi-quantitatively after secondary coding. 10

Results/Findings: Section A: Recruitment There were varying answers from the interview groups about the ways in which PSF were recruited in the community. About half the mothers believed that PSF were selected by the community or the leaders, with or without involvement of health staff, particularly in the two districts with partner support. A fifth of them believed that health staff alone selected the PSF, but more than a third (19/56) said they had no idea how the selection was made. When asked directly if they or others in the community had been consulted when the PSF were selected, less than a third said they had been (16/56). Most PSF believed that they had been chosen by the community or by the community leaders; however, almost a third of the PSF of Viqueque said the health staff had selected them. Most health workers also mentioned the community when they were asked how PSF were chosen. However, when they were asked directly to what extent the community was consulted when making the choice, about a third of them appeared to suggest that the CHC chief directly made the choice. According to most community leaders, the community was consulted in selecting the PSF, but when asked who made the selection, half of them mentioned health workers. Several criteria had been set out by the MoH to use for selecting the PSF. When health workers were asked what these criteria were, about half of them mentioned the ability to read and write and the willingness to volunteer. About a third of the health workers were not sure whether guidelines had been carefully followed during the actual selection. About half the health workers said that recruiting PSF was difficult because no salary was being offered. Both PSF and health workers were asked why PSF had agreed to become PSF. The most common answer was that it was out of a desire to help the community, or to improve the health of the community. A few PSF also said it was to be able to learn something new and develop oneself, or simply because they were chosen. Table I PSF Selection Summary Question Response Total Positive N Community How were individuals selected to be Selected by health staff 10 56 a PSF? Selected by community 18 56 Selected by health staff and 5 56 leader Selected by leaders 2 56 Don t know 19 56 Were you or other people consulted in the community? 16 56 PSF Can you explain the process of how Community Chose 28 50 you became a PSF? Community Leaders Chose 27 50 Health Staff How were the PSF in your sucos selected? Health Staff Chose 8 50 Community Leaders 15 19 CHC Chief 6 19 11

MTTs and DTTs stated that in many of their districts, health staff and local leaders had chosen their own family members to become PSF rather than consulting with the wider community. As a result, several MTTs stated that in their districts certain sucos have tense relationships between community members and PSF due to the selection process. Most responses stated that community/leaders were consulted in a majority of cases, but it is not clear if this consultation was extensive. There were clear instances where PSF were selected directly by health staff /DPHO. The main selection criterion seems to have been ability to read and write; the next most common was previous experience, either as a volunteer or in campaigns. There is no centralized list available of persons chosen to be PSF, but only of those trained. Thus, the database of trained PSF potentially underestimates the number of actual PSFs. Relative Strengths Relative Weaknesses Issue #1 Selection of PSFs The MOH has been able to recruit many PSF, at least three per suco. The majority have been chosen due to prior volunteer experience. Health staff perceive that the selection criteria are adequate. A reasonable proportion of PSF appear to be women. The majority were probably chosen by health staff without adequate community consultation, but often with involvement of community leaders. Criteria for selecting PSF were not explicitly articulated nor consistently applied. Not clear how many PSF that were recruited and then trained No clarity on replacement procedures when PSFs drop out or die -- so far replacement with PSF family members No record/reports of recruitment Recommendations for addressing gaps identified: Regular communication between health staff and local leaders on recruitment processes. Clarify recruiting processes and selection criteria for PSFs when PSFs drop out or die. Establish a database of recruitment of PSFs, updated and kept at sub-district level and compiled at National level. Consider recruiting more women PSF. Section B: Training PSF were asked what they liked and did not like about their training. Apart from a few who complained about poor facilities during training, most seem to have liked all that was taught. Trainers were generally liked, except for the occasional complaint about trainers getting angry. PSF appreciated humor and songs during training, and trainers who explained things and gave examples. When asked who had trained them, DTT and NGOs were mentioned about equally by the PSF. A much smaller proportion of health workers mentioned the NGOs, however. When asked what topics they were trained in, the PSF mentioned maternal care most often (26/50), followed by nutrition (17/50) and malaria (13/50). PSF also mentioned other topics that are not included in Module C. Knowledge about malaria probably came from other sources, but it is not 12

clear where or how. There was considerable variation among districts, suggesting that there might have been different emphasis during training in different districts. SISCa tasks were not much mentioned when this question was asked. When they were asked what they had been told they were being trained to do, small proportions of PSF mentioned the dor s or that they were told to help the community. Again, SISCa tasks were not much mentioned. When they were specifically asked if they were told to do any work other than SISCa, half of them said no such work was given to them. About half the health workers said they did not know what training was given to the PSF, and most of the rest mentioned the SISCa tasks. Only a third of them said they were present during training, however. When asked if there were still PSFs in their areas who had not been trained, 17/20 health workers said there were. Table II PSF Training Summary Question Response Total Positive N PSF What training did you receive? Pregnant/Lactating Women, 26 50 MCH Nutrition/Malnutrition/ 17 50 Complementary Feeding Malaria 13 50 Weighing 5 50 Diarrhea 5 50 Roles of PSF 7 50 Forgot 5 50 During your training what were you Help Community 11 50 told to do? Mediator 7 50 Informator 14 50 Motivator 8 50 Educator 6 50 Promoter 5 50 Where you told to do work beyond Yes 14 50 SISCa? No 25 50 It is not clear how many of the PSF were actually trained using the current, revised version of Module C, which came into effect sometime early in 2008, and how many were actually trained by the MTT/DTT teams that were organized for this purpose during 2008. Thus it is not possible to comment on the effects of the revised training material on the PSF program. A lot of training time is spent in teaching growth monitoring, but still charting and interpretation skills remain unsatisfactory. Many health workers have only a vague idea about what training was given to PSF (those who were trainers are aware). Some DTTs apparently find Module C difficult to understand. There are no complete official records of training dates, locations, and names of PSF at any level. Relative Strengths Issue #1 Training Relative Weaknesses 13

Relative Strengths The MOH was able to train over 1000 PSFs following the 6-day training schedule in Module C. The majority of PSFs liked the training and wanted more. The majority of PSFs liked their trainers. PSFs participated in the full training. PSFs received transport money during the training. Relative Weaknesses Some PSF stated that they were last trained two or more years ago, some never. There are untrained PSFs in some sucos Many have forgotten what topics they were trained in. While most of the training time is spent on training in charting and growth monitoring, these are still perceived to be the PSFs greatest weaknesses. Recommendations for addressing gaps identified: Ensure that all recognized PSF are provided one round of training using Module C and any relevant updated modules. Provide on-the-job training/mentoring at SISCa. This should be a priority over refresher trainings at the sub-district level. Use supportive supervision tools to understand and fix areas of PSF weakness Conduct refresher trainings at least once per year for PSF in addition to mentoring and supervision. MTT and DTTs suggested the MOH conduct regular refresher trainings for DTTs. Integrate SISCa into Module C (currently no mention of SISCa) Include health promotion in Module C, as this is a major skill area for PSF Consider using training time for focus on important skills, such as health promotion and interpersonal communication. Provide operational clarity during training about non-sisca activities as well Include in the training the maintenance of simple name-based records at the aldeia level, once policy decisions are made in this regard Use participatory training methods appropriate for PSF to address the poor knowledge retention as indicated in competency tests. Section C: Implementation Mothers were asked what work the PSF actually do. Most of the work recounted related to SISCa tasks weighing, registration, assisting health staff, giving people information about SISCa. A few (12/68) mentioned tasks apparently unrelated to SISCa, such as informing health workers when someone got sick, or taking care of pregnant women (but both of which may also be construed as tasks within the SISCa). The PSF themselves also overwhelmingly mentioned SISCa related activities as work they do as PSF weighing (21/49), giving information about SISCa (17/49), registration (9/49). They also mentioned health promotion (10/49) and care of pregnant women and the sick (9/49), but much of this also appears to be a part of the SISCa work. Thus, there is fair consistency between the perceptions of the community (mothers) and those of the PSF about what work the PSF do. When mothers were asked to tell of specific occasions when PSF actually attended a sick person, about a fifth of them said PSF bring medicine for sick persons, and about twice the number said the PSF provide health information of different kinds. It is not clear what medicines the PSF might bring to homes since they have no authority to give prescription medicine to the community. It may be possible they are helping deliver traditional medicine. There were few 14

concrete examples of PSF helping on specific occasions. About a fifth either said they have never seen a PSF going to people s homes or that the PSF have not yet started visiting homes. When asked to describe occasions when PSF visited homes, the commonest response was related to coming home to inform about the SISCa day or immunization day, once a month. Again, there were few examples of visiting a home for a specific health-related purpose other than related to the SISCa. Table III PSF Roles Summary Question Response Total Positive N Community What work do PSFs do? PSF What do you do as a PSF? Health Staff What health related work is the PSF expected to do? Weighing 21 56 Registration 12 56 Assist Health Staff 9 56 Inform Heath Staff is one is 8 56 sick or losing weight Give Vitamins 8 56 Share info about SISCa 7 56 Take care of pregnant 4 56 women Weighing 21 51 Registration 9 51 Give information about 17 51 SISCa Give information about 10 51 Health/Health Promotion Give information to pregnant 9 51 women and the sick Community Leaders Chose 27 51 Health Staff Chose 8 51 Talk about TB, Vitamin A, 14 19 weighing, and LILA Fill LISIO 2 19 Registration 10 19 Health Promotion 2 19 -dor 1 19 More than half the mothers had never heard of the word SISCa but clearly knew about the monthly event. About two thirds of them said such an event took place on a monthly basis in their suco. This included most mothers in Viqueque, but just 3/14 mothers in Bobonaro. Most mothers (42/48) said they had been to the SISCa at one time or another, and that they did see the PSF when they went to the SISCa. The tasks that the mothers said they saw the PSF perform at the SISCa were most often weighing and registration. When specifically asked who weighed children at the SISCa, about two-thirds of them said it was the PSF, and the rest said it was the health worker. In Viqueque, most mothers said it was the PSF who weighed children. Almost all 15

mothers said they believed the information and advice that the PSF provided, and most believed that PSF worked for all families in the suco. When the PSF were asked if they ever went to people s homes to talk to them about their health, most of them (39/51) said they did, but when they were asked who they talked to most often, only half this number (20/51) mentioned pregnant women and children. When PSF were asked to give concrete examples of how they helped specific families, a number of them stated that they encourage sick people to go to the health facility. Most PSF (40/49) said that they did health promotion. When asked where they did this, 17 said in the suco or aldeia or community, and 27 said at the SISCa. When asked what topics they talk about, about half (18/43) mentioned malaria, and fewer mentioned diarrhea (13/43), TB (9), pregnancy (9), complementary feeding (8), and malnutrition (7). In Uatucarbau sub-district, field teams conducted interviews with PSF directly after SISCa and did not observe PSF doing any health promotion. Most PSF in Aileu and Viqueque said they had received health education/promotion materials, but very few in Bobonaro said they did. All of them said they would like to have more such material, and about two-thirds said the material was useful to them in their work. Many PSF (30/50) appear to keep some written record of the work they do, although the nature of the record in not clear. They do not appear to have been provided any standard recording formats of registers. Most of them said they submit these records to the health workers. Regarding the amount of time PSF work, it is clear that most of them do attend the SISCa every month (45/48 said they had attended all the SISCa in the last three months). However, from the recorded information, it is difficult to estimate reliably how much time each PSF gives to health work other than on the SISCa day. It does appear, though, that this may not be more than a few hours every month, and much of this may be related to health campaigns such as for TT and vitamin A. When health workers were asked if they believed that the PSF were carrying out their responsibilities, all of them said they were. The health workers were asked to list the healthrelated activities that are expected to perform. While the lists they provided cover all the SISCarelated activities, each topic was mentioned by less than half the health workers (registration was the most often mentioned, by 10/20 health workers), suggesting that these expectations are not uniform or consistent. Only one health worker specifically mentioned the 4 dor s. PSFs making home visits was expected by all health workers in Aileu, but not in the other districts. Very few health workers (4/19, 3 of them in Viqueque) expected PSF to diagnose serious illnesses among children, while almost all health workers (16/19) expected PSF to educate people about health. When the health workers were asked whether PSF indeed carried out these (expected) tasks regularly, all of the said they did. This probably means that health workers are not really seeing what PSFs are doing outside SISCa, since the community states that these tasks are not regularly carried out. Community leaders were mostly positive about what the PSFs did: almost all of them said the PSF participate in SISCa regularly, share information with the community, and accompany sick people to the health facility. The majority of MTTs and DTTs in the FGDs stated that PSF should be conducting regular home visits on three main occasions: (1) to visit pregnant women to encourage women to go for ANC, (2) to visit women just after giving birth to encourage them to bring their newborn for a check-up at the health facility and promote exclusive breastfeeding, and (3) to visit sick community members to encourage them to go to the health facility rather than a traditional healer. In these instances, the PSF should also accompany the sick community member to the 16

nearest health facility rather than simply referring them. MTTs and DTTs felt that the majority of PSF know how to: register patients at SISCa, weigh children, fill out the growth chart, and do health education, but are not currently conducting home visits or referring sick children. All MTT and DTTs were in agreement that PSF play an important role in SISCa and are very helpful to health staff. Relative Strengths Relative Weaknesses Issue #1 Implementation Many PSF are active and regularly participate in monthly SISCa. Many community members are aware of PSF in their suco. The majority of community members like their PSFs. Many of the PSFs participate in campaigns (TT). Neither health workers nor community leaders felt happy about any suggestion to close down the PSF program, saying the program is good for the community, and the community would lose in terms of health information and health of the people. Recommendations for addressing gaps identified: PSF do not consistently work outside of SISCa. Health staff not clear about what training the PSF received, or about what the expected roles of PSF are. PSF want to take on more responsibilities and receive increased amounts of money. The majority of community members still do not understand the PSF program, including PSF roles, and work outside SISCa. Some PSF are helping provide medicines to homes; it is not clear how much they understand this or who is prescribing. Start regular internal communication between different program staff at the facility level (between DTTs and health workers) Continue to socialize and promote SISCa at all levels Clarify whether prescribed medicine is being provided to community members in their homes and who is prescribing or recommending it. DTTs and health staff need to monitor PSF giving out information on health topics they have not been trained in. In accordance with the guidelines, if PSFs begin to conduct home visits, then these visits require mentoring, supervision and appropriate data collection methods and tools. Section D: Supervision and Program Management Most PSF (36/47) said they go to health staff when they face a problem at work, and that the health staff are their supervisors (44/51). Some (9/51) mentioned community leaders as their supervisors, and about two thirds of them said their community leaders did talk to them about their work. None of the PSF seem to have mentioned their trainers as their supervisors. Virtually all PSF were happy with their supervisors, as they perceived them. Almost all health workers said they had written guidelines about what the PSF were supposed to do, but very few could say where these written guidelines could be found a few each said in the SISCa guidelines or in the training modules. Similarly, most of them said they had written guidelines about supporting the PSF, but it is not clear what these are. Only one of them said he had received any training in supervising the PSF. When asked what they are expected to do in 17

supervising the PSF, they all said support and counseling. Almost all of them said that besides meeting the PSF formally only at the monthly SISCa, they also met the PSF informally at other times. Community leaders were divided in their understanding about who the PSFs supervisors were: some said it was the health workers, some said it was the head of the CHC, and most of them believed that the health worker and PSF met regularly. PSF were asked whether they get paid for participating in SISCa. About three-fourths of them (37/51) said they did get paid. Those who had not yet been paid were relatively recent PSF, while the old ones were mostly paid. Of those paid, 26 were paid $5 or more at the last SISCa. The PSFs were asked if they got paid for activities other than SISCa. About half of them (26/51) said they were paid for participating in activities such as campaigns. Among the community leaders, 14/18 said the PSF did get paid for their work. A hierarchy of program management committees was expected to manage the PSF program at different levels, suco through to the national level. These were formed only in Aileu district. In none of the districts are they currently active, however, according to MTTs, other health officials and NGOs. In FGDs, MTTs and DTTs identified MTTs main roles as facilitation of training for DTTs and supervision of DTTs and facilitation of on-going PSF training, working with PSF at SISCa, and monitoring and evaluation of SISCa. MTT and DTTs stated that they are ready to conduct monthly supervision visits, but they have not received any monitoring tools. There are no guidelines about who should supervise PSF and their specific roles. However, PSF generally seem to believe that health workers are their immediate supervisors. Health workers expect PSF to do mainly SISCa related work (as is true of everyone else in the system) and do have a reasonably good idea about what tasks the PSF are or are not doing well. At least some of them appear to be supporting PSF and correcting them on the job. There is no formal structure for supervision of even SISCa activities, however. Most PSFs do not expect supervisors to provide regular support such as identifying and filling gaps in knowledge and skills but do expect their help in supplying them materials, as they would expect from authorities. There is a strong tendency to boil down all problems to need for more training and to problems with incentives either that they are inadequate or they are inadequately implemented. There is also a strong push by the MOH to strengthen monitoring and evaluation, largely in the sense of centralized monitoring. Monitoring and evaluation tools have been developed by the MoH and there are specific forms on PSF skills and performance. There are three forms: SISCa Monitoring and Stratification Form, Supervision-PSF Performance, Monitoring and Supervision Form-DPHO-Health Promotion. The first two forms have been approved for use, although only one district is currently conducting monitoring and supervision of SISCa/PSF using the national tool, as it has not yet been socialized with health staff. There is also very little vision for PSF roles beyond SISCa. Relative Strengths Issue #1 Management-Monitoring and Evaluation The MOH accomplished the large task of recruitment and training. The PSF program has covered all 13 Relative Weaknesses PSF, health staff and community leaders are unclear about their roles. No supervision structure 18

Relative Strengths Relative Weaknesses districts in a short time span. No supervision tools The management structure described in the guidelines has the potential to be Health staff have not been oriented on how to supervise PSF. effective. Funding for SISCa is inconsistent. SISCa provides a solid structure to manage PSFs. Poor internal communication amongst heath staff, DTTs and non-dtts Community leaders are common supervisors but have no technical knowledge to support PSF. Recommendations for addressing gaps identified: Create a transparent system to manage and record PSF monthly payments Make it a priority to guarantee timely monthly payments to PSF at SISCa Consider adding more details for money management at SISCa, because current guidelines are quite broad Clarify roles and responsibilities from the national level to the health post level regarding supervision of PSF: what should each level be doing? Socialize supervision tools for all health staff Conduct regular monitoring and supervision Create a database from the national to sub-district level that includes: PSF names, training dates, training locations, facilitators per each training, gender of PSFs, records of PSF payments, PSF names per SISCa post Provide clear supervisory guidelines and tools to health workers, and stop expecting MTT/DTT to oversee the program, supervision schedules Rationalize incentive structures, create a recording system for payments, develop guidelines for SISCa/PSF funds Currently there is no system at aldeia level that captures information on service coverage or household-level behaviors. A simple name-based census and service register, maintained by PSF and community leaders, could routinely capture this information. These can be brought to the SISCa monthly, so that the health worker can update her/his own registers, and thus keep the entire HMIS updated. This will also create enable accurate tracking for preventive services as well as referral. Section E: Program Strengths and Limitations, as perceived by different stakeholders Mothers were asked what they liked and did not like about PSF in their communities. The commonly mentioned aspects that they liked included getting useful health-related information from the PSF, bringing medicines to them, taking patients to the hospital and taking children for weighing. Most mothers found nothing about the program that they did not like. Most believe that the PSF are useful in solving their health problems, and most said that they did not know of anyone other than PSF who could solve their health problems. Virtually all mothers believe that the PSF like the work they are doing, although a few of them mentioned that they do not like it when they do not get paid. When health workers were asked the same question, most of them (14/18) also said they believed the PSF were happy with their work. When PSF were asked what they enjoyed about being a PSF, they most commonly said they liked helping the community and giving information. Also, most of then had nothing to say 19

when asked what they did not like, but a few did not like not getting to eat anything at the SISCa, some did not like having to walk all the way to the SISCa, a few complained about lack of support from community leaders and from the government. When they were asked what their spouses think about their being PSF, most of them said their spouses were happy (25/47) or supportive (13/47). Only two PSF, both males, said their spouses were unhappy. Five PSF said their spouses were worried because of lack of incentive. Almost all health workers (17/19) believe that having PSF has made a difference to the health work they do. Most community leaders believe PSF have benefited their suco, and most often mentioned sharing of information as the manner in which they have benefited the communities. Other commonly cited reasons are that, with the PSF around, mothers and children have easy access to health facilities, and that the PSF are useful assistants to health workers. Health workers were asked to say whether PSF were good or bad at each of a list of tasks. The opinion of most of them was that the PSF were good at registration (16/18) and weighing (17/18). Fewer said they were good at health education (14/18), plotting (11/18) and interpreting growth charts (13/18), and at referring sick children to facilities (12/18). Only about a third (7/18) said they were good at making home visits. About half the health workers said PSF attend to health problems of all families in the village, but almost all of them felt that PSF have adequate knowledge of health and disease, that they get adequate support from health staff and that people in communities are happy with them. Table IV Strengths and Limitations Summary Question Response Total Positive N Community What do you like about having PSF s in your community? PSF What do you not like about being a PSF? They bring patients to the hospital They bring medicine to the community Share information concerning health They bring children for weighing 10 58 18 58 5 58 2 58 Nothing 22 50 Walking to/reaching SISCa 5 50 Not getting anything to eat at 5 50 SISCa No support from community leaders 3 50 No attention from 4 50 MOH/Government Perceptions about what can be done to improve the PSF program: About a third of mothers believed that offering better incentives to PSFs will be useful. A similar proportion of PSFs (14/47) and health workers (5/18) felt the same. 20

PSFs also felt that better work facilities were needed (such as chairs to sit on during SISCa), and more training and more educational materials. A small proportion of them (7/47) wanted training in case management. Health workers said the PSF need more training, particularly on malaria, diarrhea, and nutrition. Neither health workers nor community leaders felt happy about any suggestion to close down the PSF program, saying the program is good for the community, and the community would lose in terms of health information and health of the people. MTT and DTTs feel strongly that the 35 dollars for SISCa needs to have transparent mechanisms and should be guaranteed monthly. MTT/DTTs suggested that the MOH provide identification cards for all PSFs and organize regular special events for PSFs such as district to district study tours or competitions. This would make PSFs feel official in their roles as PSFs and recognized by the MOH. Heads of departments and program official that were interviewed at the national level felt that there should be a coordinator for MTT/DTTs so that they are managed closely; and that management committees for the program should be formed and meet regular so that xefi sucos are strongly involved in the PSF/SISCa programs, clarify PSF roles and money and regular monitoring needs to be conducted. Section F: Competency Tests Competency tests were given to PSFs, mothers, health staff and MTT/DTTs. The tests administered to the PSF as well as to mothers of children who were interviewed consisted of simple knowledge and awareness questions. A simple exercise also tested PSFs charting skills. Similar tests were also administered to health staff. The scoring of the tests followed a simple pattern: a completely correct answer got a score of 2, a partially correct answer got a score of 1 and a totally wrong answer got a zero. The total score for an individual was a simple sum of the scores of individual questions for that person. Since the number of questions on the knowledge tests was not even, total achieved scores are expressed as simple percentages of the maximum possible total score. Seven questions were common to the competency tests administered to mothers and to the PSF. These were scored separately to get a sense of how lay persons, who were presumably exposed to some health education over time, compared to the PSF, who were also from the same communities, but had been formally trained. Despite the small numbers involved, an attempt was made to compare knowledge/competency scored over relevant subcategories. Komunidade test scores 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0 SISCa Isin Rua Susuben Fase liman Nutrisaun Immunizasaun 21

PSF test scores 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0 SISCa_PSF Isin Rua Susuben Diarea_fase liman Nutrisaun Immunizasaun The total score percent among the PSF was 55.3%, which was marginally better than that among mothers, which was 50.0%. Health workers scored higher, at 72.8%. Among comparable questions, mothers scored 7.8 out of a possible 14, while the PSF scored a little more, at 8.6. There were small differences across districts, the PSF in Viqueque scoring about 1.8 points more than mothers, while the smallest difference was 0.3 points in Aileu. Most of the difference between mothers and PSF appeared to be in one sub-district, Uatucarbau of Viqueque here, PSF scored 2.5 points more than mothers. In terms of total scores, the PSF in Lolotoe and Uatucarbau sub-districts appeared to have performed distinctly better than mothers, with percentage point differences of 12.5 and 10.6, respectively. The other variable that appeared to make some difference was the gender of the PSF: female PSF scored 2 points more than male PSF on the average. Since Viqueque (and Uatucarbau subdistrict) had the smallest proportion of women among PSF, the better performance of the PSF in this district does not appear to be related to the gender distribution. Comparing female PSF to mothers, the difference in comparable knowledge scores was 2 points, and the difference in total scores was 8.7%. PSF who had previous experience as health volunteers did not do better than those without experience. The information available from the interviews and from the available databases related to PSF training are insufficient to identify with certainty how many of the interviewed PSF were actually trained in the last one year (since the beginning of 2008), and thus it is not possible to comment on the influence of the current training on PSFs competency test scores. However, PSF did report on the length of time they consider having been a PSF, and thus it is possible to differentiate between new recruits (of 2008) and recruits from periods previous to this. There are only minor differences, with the older PSF faring better than those who report to be more recent. 22

Pessoal Saude test scores 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0 SISCa_PSF Isin Rua Susuben Diarea_fase liman Nutrisaun Immunizasaun Although health workers overall scored better than community members or PSF on the same questions, on average they had similar strong and weak areas of knowledge as the PSFs. This included good knowledge about time to complete immunization and good knowledge on exclusive breastfeeding, colostrum, diarrhea management and complementary feeding. Knowledge about the roles of PSF, SISCa schedules, hand-washing and about antenatal care was lower. Nutrition knowledge was reasonable but specific counseling advice for malnourished children was generally poor. DTT & MTT test scores 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0 SISCa_PSF Jestaun Isin Rua Susuben Diarea_fase liman Nutrisaun Immunizasaun DTTs and MTTs also showed a similar pattern of strengths and weaknesses in knowledge. Immunization was known perfectly and breastfeeding, diarrhea and nutrition knowledge was again reasonable. Nutrition scores were weaker, however, because of poorer skills in counseling for malnourished and marking the growth chart. Knowledge of SISCa schedules and PSF roles was moderate. This was mostly because of inconsistent understanding of the hours that PSFs should commit to each month. Antenatal care knowledge was weaker, and there was poor understanding of the management and reporting structure for the PSF program. This is almost certainly because it has been established only in very few sucos. The timing requirements for reporting were not known because regular reporting between levels has no been established. Additional Insights from Community Members Perceptions about health problems: 23