Youth-Friendly Services End of Program Evaluation Report. African Youth Alliance (AYA)

Transcription

1 Youth-Friendly Services End of Program Evaluation Report African Youth Alliance (AYA) November 2005

2 Table of Contents: List of Acronyms... i EXECUTIVE SUMMARY... ii INTRODUCTION... 1 OVERALL METHODOLOGY... 5 STATIC FACILITY EVALUATION... 8 Static Facility Activities... 8 Facility Reassessment Evaluation Methodology Results Analysis of Training Data Evaluation Methodology Results Analysis of Client Satisfaction Data Evaluation Methodology Results Trend Analysis Evaluation Methodology Results OUTREACH EVALUATION Outreach Activities Analysis of Training Data Evaluation Methodology Results Analysis of Client Satisfaction Data Evaluation Methodology Results Trend Analysis Evaluation Methodology Results Analysis of Monitoring Data Results CONCLUSIONS AND RECOMMENDATIONS... 61

3 List of Acronyms AYA ASRH BCC CRHW CSW DISH FLEP HC HIV/AIDS KAYA KCC NGO PATH QOC SRH SRD STI UNFPA UYDEL VCT YFS African Youth Alliance Adolescent Sexual and Reproductive Health Behavior Change Communication Community Reproductive Health Worker Commercial Sex Worker Delivery of Improved Services for Health Family Life Education Project Health Center Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome Kampala Youth in Action Kampala City Council Non-Governmental Organization Program for Appropriate Technology Quality of Care Sexual and Reproductive Health South Rwenzori Diocese Sexually Transmitted Infection United Nations Population Fund Uganda Youth Development Link Voluntary Counseling and Testing Youth-Friendly Services i

4 EXECUTIVE SUMMARY The African Youth Alliance (AYA) was launched in the fall of 2000 by Pathfinder International, the Program for Appropriate Technology in Health (PATH), and the United Nations Population Fund (UNFPA). Since its inception, AYA s objective has been to improve overall Adolescent Sexual and Reproductive Health (ASRH) and reduce the spread of HIV/AIDS and other Sexually Transmitted Infections (STIs) in four African countries: Botswana, Ghana, Tanzania, and Uganda. Each of the three founding AYA partners brought their own unique expertise to the five-year project. Pathfinder International s contribution in each of the AYA countries was the development and expansion of Youth-Friendly Services (YFS) and institutional capacity building. This report highlights the results of the YFS work that was implemented by partners 1 in 13 districts in Uganda including: Kasese, Kabarole, Kyenjojo, Kabale, Kapchorwa, Mbale, Soroti, Kaberamaido, Sironko, Iganga and Mayuge. The AYA/Pathfinder approach to YFS focused on the following: Building on existing resources, using available facilities and service providers Reaching young people through a variety of channels such as: static clinics, outreach including peer education, and the private and commercial sectors Establishing linkages with effective referral sites Creating partnerships with other institutions for future scaling-up Instituting a minimum package of youth-friendly Sexual and Reproductive Health services (SRH), including: o Information and counseling on sexuality, safe sex, and reproductive health o Contraceptive method provision (with an emphasis on dual protection) o STI diagnosis and management o HIV counseling (and referral for testing and care) o Pregnancy testing and antenatal and postnatal care o Counseling on sexual violence and abuse (and referral for needed services) o Postabortion care counseling and contraception (with referral for treatment of complications when necessary) Specifically, the AYA/Pathfinder strategy for institutionalizing youth-friendly SRH services included the following: Facility assessments Development and implementation of action plans for quality improvements based on the results of the facility assessments Provision of essential technical assistance and monitoring to the institutions, management and clinics as per identified facility strengthening needs Training of service providers in ASRH/YFS 1 The partners included 14 NGOs and districts throughout the country. The NGO partners and the districts they served included: Kampala City Council (Kampala), UYDEL (Kampala), Busoga Diocese (Iganga and Mayuge), South Rwenzori Diocese (Kasese). The district partners included: Mbale, Soroti, Kaberamaido, Kabale, Kabarole, Kamwenge, Kyenjojo, Kapchorwa, and Sironko. An additional partner, Makerere Medical School, was chosen for the pre-service training work done under this project. ii

5 Assistance on data collection and analysis of service statistics Implementation of youth input and feedback mechanisms Creation and/or expansion of peer education programs Community sensitization in SRH and involvement in peer selection for outreach work Institutionalization of YFS through the development of standards and guidelines, YFS tools, and YFS curricula for in-service and pre-service training AYA/Pathfinder achieved the following results (organized by static facility and outreach results): Static Facility Results Facility Reassessment: Reassessments were carried out at the end of the project in selected facilities that showed improvement in the five elements of youth friendliness assessed in the facilities: privacy ensured, competent staff, minimum package of services, peer providers/counselors available, and publicity for YFS. The greatest improvement was seen in the availability of peer providers and counselors, followed by the availability of a trained service provider in the facility to serve young people. Analysis of Training Data: 202 service providers were trained and all showed increased scores from pre- to post-tests. The comparison of pre- and post-tests from 10 trainings showed an average gain in scores ranging from 9 to 30, with an average gain of 20 points for all. Analysis of Client Satisfaction Data: Mystery client data were analyzed from three of the five districts that conducted visits. The clients that visited the clinics reported being satisfied with the services provided. They noted that the waiting time was reasonable and privacy was observed. The service providers were youth friendly and demonstrated good communication skills, and overall the services were rated as affordable. The use of Behavior Change Communication (BCC) materials was limited in these cases, but was likely due to shortages of supplies. Overall satisfaction with the services received from the clinics was well demonstrated by the client feeling confident to recommend the service provider to a friend. Trend analysis: From January 2003 to December 2004, 195,591 clinic visits by youth were recorded. More visits were made by youth years of age than other age groups and females made more visits than males. Females years made the most visits, followed by females 15-19, males 20-24, and males Youth sought counseling more than other available services (family planning services, STI testing and treatment, and pregnancy-related services). Despite increases in the youth-friendliness of clinics, trend analysis shows an overall decrease in visits to the facilities over the project period. 2 2 It should be noted that because reporting was collected by partner (as opposed to by facility), and not all facilities reported each quarter, the numbers of facilities reporting by quarter may not be consistent. Therefore, the number of visits may not have gone down, but rather this could be a reflection of a decrease in the number of facilities reporting. iii

6 Findings also revealed that new visits to facilities exceeded revisits and condoms were the most preferred contraceptive method for clinic youth. Outreach Results Analysis of Training Data: The national trainers trained over 647 peer providers. The information introduced increased scores from pre- to post-test. In a review of 14 training workshop results, the average marks gained from pre- to post-test for each workshop ranged from 11 to 43, giving an average gain of 23.6 for all the trainings. Analysis of Client Satisfaction Data: Mystery client visits with peer providers in four districts were analyzed to gauge client satisfaction with peer provider services. There was overall satisfaction with the friendliness and communication skills of the peer providers. The peer providers served the youth well, by distributing supplies, making referrals, and keeping records. Overall satisfaction of the services received from the peer providers was well demonstrated by the client feeling confident to recommend the provider to a friend. Trend Analysis: Between January 2003 and December 2004, records reflect 580,771 outreach visits made by youth. More visits were made by youth years of age than other age groups and more by males than females. Males made the most visits, followed by females years, females years, and males years. Male condoms were the most sought after contraceptive from the peer providers. Analysis of Monitoring Data: Analysis of monitoring reports found that peer providers faced a number of challenges in their work. Challenges included the use of in-school youth who lived and went to school in different districts, lack of transportation to cover the long distances of their coverage areas, and shortages of male condoms. Overall, the AYA/Pathfinder program was found to be successful. However, future efforts such as these could be strengthened by the following recommendations: future programs should allocate more staff time and funding for supervision, document program processes more effectively, build evaluation processes into the program at the beginning, and include efforts to improve contraceptive supply. iv

7 INTRODUCTION The African Youth Alliance (AYA) was launched in the fall of 2000 by Pathfinder International, the Program for Appropriate Technology in Health (PATH), and the United Nations Population Fund (UNFPA). AYA sought to improve overall Adolescent Sexual and Reproductive Health (ASRH) and reduce the spread of HIV/AIDS and other Sexually Transmitted Infections (STIs) in four African countries Botswana, Ghana, Tanzania, and Uganda. The main beneficiaries for the project were young people between the ages of 10 and 24. The secondary targets included teachers, health workers, social workers, and parents. In addition, the tertiary target group included religious leaders, media workers, politicians, and policy makers. The latter group was crucial for creating an enabling environment for the project. The project was developed with a focus on six broad areas, including: 1) Advocacy and policy: The creation of supportive community and political environments through policy and advocacy efforts at both the national and community levels, and efforts to improve communication between young people and the adults in their lives. 2) Behavior Change Communication (BCC): The development and expansion of behavior change communication through interpersonal communication; folk and mass media, including drama; life planning skills programs for youth; peer education and counseling; and social marketing campaigns. 3) Youth-friendly services: The improvement of young people s access to and the quality of reproductive health services by developing, expanding, and institutionalizing youth-friendly services in a variety of settings. 4) Institutional capacity building: Strengthening the institutional capacity of the countrylevel partners so they can better plan, implement, manage, and sustain programs and services. 5) Life and livelihood skills development: The integration of sexual and reproductive health into existing livelihood skills development and training programs for youth. 6) Coordination and dissemination: Coordination and information sharing of program activities, lessons learned, and best practices. Pathfinder International was responsible for the Youth Friendly Services (YFS) and institutional capacity building components implemented in each country. Through the YFS component, AYA/Pathfinder sought to address the factors that hinder young people from seeking services and to make them youth-friendly. Youth-friendly services are those that attract youth, meet a variety of young people s needs comfortably and responsively, and succeed in retaining them for continuous care. Pathfinder had, through previous work worldwide, developed a list of the key elements of youth-friendly services. Under AYA, these have been categorized into essential and supportive elements as presented in table 1. 1

8 Table 1: Characteristics of Youth Friendly Services Essential Supportive Convenient open hours Privacy ensured Competent staff Respect for youth Minimum package of services available Sufficient supply of commodities and drugs Range of family planning methods offered Emphasis on dual protection/condoms Referrals available Young adolescents (12-15 years-old) are served Confidentiality ensured Waiting time not excessive Affordable fees Separate space and/or hours for youth Youth input/feedback to operations Accessible location Publicity for YFS Comfortable setting Peer providers/counselors available Educational materials available Delay of blood test and pelvic exam, if possible Partners welcomed and served Nonmedical staff oriented Provision of additional educational opportunities Outreach services available The AYA/Pathfinder approach to YFS focused on the following: Building on existing resources, using available facilities and service providers Reaching young people through a variety of channels such as: static clinics, outreach including peer education, and the private and commercial sectors Establishing linkages with effective referral sites Creating partnerships with other institutions to sustain efforts Instituting a minimum package of youth-friendly Sexual and Reproductive Health (SRH) services, including: o Information and counseling on sexuality, safe sex, and reproductive health o Contraceptive method provision (with an emphasis on dual protection) o STI diagnosis and management o HIV counseling (and referral for testing and care) o Pregnancy testing and antenatal and postnatal care o Counseling on sexual violence and abuse (and referral for needed services) o Postabortion care counseling and contraception (with referral for treatment of complications when necessary) 2

9 AYA/Pathfinder s YFS work is reflected in the conceptual framework presented below (fig. 1). Figure 1: Uganda s YFS Conceptual Framework INPUTS OUTPUTS EFFECTS FACILITY ASSESSMENT ACTION PLAN TRAINING OF SERVICE PROVIDERS FACILITY STRENGHTENING PRE-SERVICE IN-SERVICE CURRICULUM AND STANDARDS AND GUIDELINES TRAINING OF OUTREACH STAFF COMMUNITY & STAKEHOLDER MOBILIZATION SERVICE QUALITY IMPROVEMENT CLIENTS Counseling Services Behavior As shown above, the AYA/Pathfinder strategy for institutionalizing youth-friendly SRH services included the following: Facility assessments Development and implementation of action plans for quality improvements based on the results of the facility assessments Provision of essential technical assistance and monitoring to the institutions, management and clinics as per identified facility strengthening needs Training of service providers in ASRH/YFS Assistance on data collection and analysis of service statistics Implementation of youth input and feedback mechanisms Creation and/or expansion of peer education programs Community sensitization in SRH and involvement in peer selection for outreach work Institutionalization of YFS through development of standards and guidelines, YFS tools, and YFS curricula for in-service and pre-service training AYA/Pathfinder began its work by presenting the AYA project in one-day workshops at several district headquarters. During these workshops the team (with representation from PATH, UNFPA, and Pathfinder International) discussed the criteria for selecting partners to implement youth-friendly services. Partners were selected that: already implemented youth development initiatives/programs 3

10 had national or regional coverage and whose technical capacity could be developed and/or strengthened to undertake or replicate national programs had experience in managing donor funded programs involved youth in programming worked with or were linked to government programs showed a desire for sustainability of the program showed willingness to implement monitoring and other systems to systematically measure progress and impact In Uganda, the AYA/Pathfinder work was implemented by 14 partners, including nongovernmental organizations (NGOs) and districts throughout the country. The NGO partners and the districts they serve included: Kampala City Council (KCC) (Kampala) Uganda Youth Development Link (UYDEL) (Kampala) Busoga Diocese (Iganga and Mayuge) South Rwenzori Diocese (SRD) (Kasese) The district partners included: Mbale Soroti Kaberamaido Kabale Kabarole Kamwenge Kyenjojo Kapchorwa Sironko An additional partner, Makerere Medical School, was chosen for the pre-service training work. This report highlights the results of the YFS work implemented in Uganda. It describes the work implemented by AYA/Pathfinder staff in the country, the process used to evaluate the interventions, and the findings of the evaluation. It also offers recommendations on implementing and evaluating YFS efforts. 4

11 OVERALL METHODOLOGY The YFS evaluation consisted of a number of activities designed to assess the extent to which the interventions met their objectives (increased use of services), as well as to capture successes, challenges, and lessons learned of both facility and outreach efforts. The evaluation process was designed by both Pathfinder International headquarters and field staff and implementation was carried out by the field staff, with assistance from Pathfinder headquarters. Key evaluation activities included: evaluation planning facility reassessments analysis of training data analysis of client satisfaction data trend analysis analysis of monitoring data The diagram below shows the evaluation activities under the static facility and outreach efforts, forming the outline for this report. Each of the activities is described generally in this section and then more specifically as it relates to the facility and outreach evaluations later in the report. Figure 2: Uganda Evaluation Framework Evaluation Planning STATIC FACILITY EVALUATION OUTREACH EVALUATION Facility Reassessment Analysis of Training Data Analysis of Client Satisfaction Data Analysis of Training Data Analysis of Client Satisfaction Data Trend Analysis Analysis of Monitoring Data Evaluation Planning: An evaluation strategy meeting was conducted in November 2004 with headquarters and field staff. As part of the design process, Uganda listed its major intervention areas and then weighted these in relation to the level of effort invested (time, human resources, and money). 5

12 Based on this information, staff selected the following key activities to evaluate based on available resources: facility strengthening, training, and client satisfaction. Facility reassessments: AYA/Pathfinder field staff reassessed a sample of facilities using the facility assessment tool, 3 and applied the certification tool 4 to establish endline results in January These results were compared against the baseline scores obtained at the outset of the project. It should be noted that the original baseline information obtained through the facility assessment tool was qualitative in nature and was intended primarily for planning purposes. In order to quantify the baseline, a retroactive scoring process was used whereby a quantitative scoring tool (i.e., the certification tool) was applied to the facility assessment results to obtain a numerical score. Essential and supportive elements were scored as follows: Score 2: If the element meets the criterion fully Score 1: If the element meets the criterion partially or if actions are underway to comply Score 0: If the element does not meet the criterion Analysis of training data: Trainings were evaluated using a combination of pre- and posttest assessments and an end of training evaluation. Pre- and post-test data were analyzed in January The average scores gained per training were calculated and comparisons were made across the different types of trainings. A review of training evaluation summaries and discussions with national and district trainers provided additional information. Analysis of client satisfaction data: Youth served as mystery clients in order to gauge client satisfaction with service provision at the clinics and from peer providers. Interview results of 99 mystery client visits to clinics and 101 visits with peer providers between April and November 2004 were analyzed in July Trend analysis: Each facility and peer provider collected and reported service statistics on a quarterly basis. A trend analysis of those data was conducted in April 2005 to reveal changes in the service statistics following the YFS intervention. At the November 2004 evaluation strategy meeting, AYA/Pathfinder staff agreed to examine trends in the following indicators: Number of visits (new and revisit) Number of visits by age (10-14, 15-19, 20-24) and sex Voluntary Counseling and Testing (VCT) 3 The facility assessment tool, Clinic Assessment of Youth-Friendly Services: A Tool for Improving Reproductive Health Services for Youth, can be downloaded from Pathfinder International s website at 4 The Certification Tool for Youth Friendly Services can be downloaded from Pathfinder International s website at 6

13 Analysis of monitoring data: An analysis of monitoring data, including peer provider and supervisor reports, was done to provide additional information for this report. Overall Data Limitations There were a number of limitations to the data and the evaluation itself, including lack of service statistics, lack of analysis of data, retroactive scoring, staff turnover at the end of the project, and lack of funding. Service statistics: AYA/Pathfinder was unable to obtain monitoring data from all of its partners, including facility and peer provider service statistics. For example, Busoga Diocese, a partner who implemented YFS in January of 2003, including a large outreach component, did not provide statistics of its work after March 2004 (despite numerous attempts on the part of AYA staff). In addition, the district partners began in late 2003, so their reported statistics only covered one year. Finally, despite attempts to work with the implementing partners to collect and clean the data, some data were not able to be fixed and could not be analyzed, thereby decreasing numbers reported. Lack of analysis of data: Another limitation was the lack of analysis and explanation of improvements seen throughout the project and reported in the results, caused by a lack of resources. For example, specific information was not captured on why facilities made improvements or why increases or decreases in services occurred at particular times throughout the project. Despite attempts to obtain this information from project staff throughout implementation, this information is missing. Retroactive scoring: The retroactive scoring process was a significant limitation to the reassessment data because many baseline assessments were scored months after the initial assessment, which was done without the certification tool in mind. This meant that in some cases, related information required to score appropriately was missing. Though efforts were made to find missing information, these were not always successful. Because the endline assessments were done after the development and application of the certification tool to baseline scores, endline assessment information was found to be more complete. Staff turnover: Unfortunately, due to the upcoming project end date, two key Pathfinder staff left the AYA project in November and December 2004, and were not able to complete the evaluation activities as planned. As a result, much of the work had to be done by consultants that had some familiarity with the project and by new staff. These circumstances have limited some detail and information in the report. Lack of funding: As often happens as multiyear projects near an end, funding limitations affect end of project activities and particularly evaluation work. The evaluation design was scaled back in this case to provide the best information possible given the resources available. The following section discusses the static facility evaluation. 7

14 STATIC FACILITY EVALUATION This section describes the activities done under the static facility component and describes the results of the evaluation of the static facility work, including facility reassessment, analysis of training data, analysis of client satisfaction data, and trend analysis. For each section the methodology for evaluation, data limitations, and results are provided. Static Facility Activities The Uganda static facility component consisted of the following activities: Development of a national ASRH training team Development of a national training curriculum Training of district trainers Selection and training of assessment teams Facility assessments Selection of clinics for facility strengthening Development of quality improvement action plans Implementation of action plans Monitoring and supervision Use of mystery client visits and exit interviews Development of a national ASRH training team AYA/Pathfinder supported the development of a national ASRH training team to conduct trainings for the health workers in the 13 AYA districts. A consultant from Pathfinder trained the team of 20 national trainers in May The training was designed to improve the participants capacity to plan, conduct, and evaluate trainings, and addressed various aspects of training management. The training of trainers was followed by a seven-day training in ASRH/YFS in July and August. Several previously trained trainers from Botswana participated in the ASRH/YFS training, which provided an opportunity for skills sharing among AYA countries. Development of a national training curriculum The national trainers were involved in the review of the national training curriculum. The objectives of the review were to: Identify issues and suggestions for improving curriculum; Solicit suggestions and recommendations from participants on how to improve the document; Enable participants to apply knowledge and skills acquired during the master trainer training; Familiarize national trainers with the ASRH training materials. 8

15 Training of district trainers Twenty-eight people (eight male and 20 female) from 10 AYA districts participated in a seven-day training on ASRH. They included three clinic officers, two public health nurses, three youth, seven nurse midwives, three service delivery coordinators, one youth officer, six staff of the Department of Gender and Community Development, one from the Department of Education, a district health provider, and a health visitor. Selection and training of assessment teams The national trainers also supported the facility assessment exercise. The team participated in a one-day training on assessing and planning for youth-friendly services; this prepared them to take part in the exercise as facilitators, assessors, and supervisors of data collection and report writing. The training covered characteristics of YFS, assessment methodologies, challenges to the assessment process, and a review of the YFS tool. The national trainers conducted facility assessment trainings for the district teams. The first district training was a one-day training in the Kasese district. The trainers noted that one day was not enough for the facility assessment training, so later trainings were increased to two days in the other districts. The additional day allowed sufficient time for the teams to effectively cover topics such as assessment challenges and how to conduct a participatory debriefing session. Facility assessments After the trainings, district teams were divided into groups of four to six people (one national trainer, two to three district officers and one to two young people) to conduct baseline facility assessments. Each assessment took one day. The numbers of facilities assessed varied from district to district. At the conclusion of each facility assessment, the team prepared a debriefing note, which was used to facilitate discussion of the findings with the facility staff. This exercise enabled the team to highlight both positive points as well as areas where improvement was required. This also allowed facility staff to provide feedback and their own recommendations for improvement. The facility report summaries were shared with the partners. Baseline assessments were conducted in 154 facilities between August 2002 and March 2003 in the districts of Kasese, Kabarole, Kabale, Iganga, Mayuge, Mbale, Soroti, Kaberamaido, Kampala, Kamwenge, Kyenjojo, Kapchorwa, and Siroko from August 2002 to April

16 Producing quality improvement action plans Although the assessment teams planned to create action plans on the last day of the assessment, almost none had sufficient time to do so. In the Kasese district, a team of national trainers shared facility assessment reports with facility managers in a two-day workshop and came up with draft facility action plans. These action plans were later refined by the South Rwenzori Diocese staff of the Young and Powerful Initiative. For other partners, action planning at the facility level was not done as a result of changes in management and delays. Instead, plans were produced at the district level for all of the facilities in that district in a systems-wide means using summaries of the actions required for improvement in each facility. Selection of clinics Although 154 clinics were assessed, clinics had to then be selected for implementation of YFS. In the first districts where assessments were conducted (Kasese, Iganga and Mayuge), the district assessment teams assessed all the facilities that were presented by the partners. However, the assessment teams realized that implementation would not be possible in all the clinics being selected by the partners due to a shortage of resources, and that clinics would need to meet additional criteria in order to be assessed. Thus, in the remaining districts, the national teams first interviewed the district officials about the facilities to establish if they were located in the AYA selected health subdistricts, if they were in subcounties where BCC activities were taking place, and if they were located near places where young people congregate. For example, Kasese had 22 facilities assessed, but only nine were selected for implementation based on facility selection criteria. In the remaining districts, the number of facilities assessed ranged from 5 to 12 facilities and almost all were selected for implementation. In the end, 96 facilities were selected for implementation. In 20 of those facilities, implementation was limited to training (see Appendix A). Implementation of quality improvement action plans Facility action plans focused on making physical improvements (such as painting and partitioning to increase privacy and make clinics more youth friendly), creating linkages (between facilities of different levels 5 and between facilities and outreach/community 5 Uganda has several facility levels: Health Centre I is a community level facility, which does not usually contain physical infrastructure, but is for planning purposes. It includes community health workers of all types (i.e., outreach workers). Health Centre II contains curative and preventive services (i.e., immunizations, treatment of minor ailments) with no surgical interventions and can provide simple family planning services and syndromic treatment of STIs (no laboratory facilities). It is usually staffed with only a nurse and a nursing assistant, but in cases where there is a midwife, it can provide ANC and normal delivery services. HC III is at the sub-county administrative level and provides both curative and preventive services, including minor surgery under local anesthesia. HC111 are staffed by nurses, nursing assistants, midwives, clinical officers, and there are provisions for simple lab tests, family planning, ANC, normal deliveries, STI prevention and treatment, ASRH services, assisted delivery (with vacuum 10

17 providers and activities), and training of service providers. Service provider training was the primary activity implemented in Uganda, and was done at two levels: in-service and pre-service. In-service training Districts selected a minimum of six service providers at least one from each of the YFS facilities to be trained and subsequently provide YFS in their facilities. The participants included clinical officers, registered nurses/midwives, enrolled nurses/midwives and a few nursing assistants. Participants were selected that had a minimum education level of O-level (four years of secondary school) and who had previous training in reproductive health. A total of 202 service providers were trained throughout the project period. The trainings focused on equipping service providers with knowledge, skills, and positive attitudes on adolescent, sexual, and reproductive health. Training of clinical service providers was conducted using the Ministry of Health s National Training Curriculum for Health Workers on Adolescent Health and Development. During the two-week trainings, the service providers were taken through six modules covering: Introduction to training Communication and counseling for adolescents Life skills, gender and culture Adolescent sexual and reproductive health Providing youth-friendly health services Drug and substance abuse Behavior change communication An overview of AYA and adolescent health issues in Uganda were also included in the trainings. Pre-service training AYA/Pathfinder also integrated YFS training in the pre-service training for medical professionals in collaboration with the Makerere Medical School. A total of 18 tutors (at least two from each of the participating schools and eight from Mulago) were trained. In addition, 48 health workers (nurses and doctors) and 17 preceptors were trained, equipping them with knowledge and skills to support students in service provision to youth during their practical rotations. extraction), PNC, and basic emergency services. Ideally should also be able to provide IUD insertion, Norplant, MVA for PAC and I/V fluids, but that is often dependent on the skills on the midwife available. HCIV is the first referral level and has all of the services outlined for HCIII, but also has a resident medical officer and an anesthetic assistant, so should also be able to provide emergency surgery (i.e., Ceaserian section), surgical contraception and blood transfusions. 11

18 Through several sessions, the tutors collectively developed the curriculum with similar content for the different training programs. They designed a module-based curriculum, a curriculum guide, and a student manual. Modules 1-3 of the curriculum were piloted in all schools in August and September 2004, however, modules 4-6 could not be covered during the school s calendar year. Monitoring and Supervision Monitoring and supervision was carried out throughout the project in order to identify weak areas and make improvements to the facilities and project implementation as necessary. Monitoring and supervision included collection and analysis of facility service statistics and regular supervision by district and partner staff, as well as AYA/Pathfinder staff, as described in more detail below: Collection and analysis of service statistics: Facility service statistics were collected and monitored throughout the project period. Facilities were provided with data collection forms, which disaggregated data by sex, age (10-14, 15-19, 20-24), type of visit (new or revisit 6 ), and services provided to each client. The data were compiled and submitted to AYA/Pathfinder on a quarterly basis by the partners. AYA staff reviewed the statistics on an ongoing basis for purposes of strengthening of data collection and of implementation. Supervision by AYA/Pathfinder and district and partner staff: Supervision of AYA/Pathfinder staff included visits to facilities to review progress on planned activities, assess linkages with other AYA activities in the area, and inspect the conditions of the facility environment. The district supervision teams from the office of the director of District Health Services conducted monthly supervision visits in which they looked at the services being provided, availability of supplies and procurement plans, the records prepared by the service providers, and progress on other planned activities. All partners shared their experiences and reviewed progress made under AYA during annual meetings. Use of mystery client visits and exit interviews Exit interviews and mystery clients 7 were used to assess client satisfaction at Mbale, Kabale, UYDEL, KCC, and SRD facilities. In order to gather information on the quality of service provision, exit interviews were conducted with youth as they left the facilities. Mystery clients were chosen by partner and district officers, with the help of peer providers. They could be male or female but in order to be representative of the majority of young people in the community, could not be working with the AYA program. 6 If the client was new to the facility, the visit was marked as new. If the client had been served at the facility previously, regardless of what they were served for, the visit was marked as revisit. 7 Mystery clients are trained people (usually community members) who visit program facilities in the assumed role of clients, and then report (by completing a survey or through an interview) on their experience. 12

19 Training was carried out for the selected mystery clients. The content of the training included introducing the participants to the overall purpose of the exercise, the steps involved, and what they would be looking for. The mystery clients were also given the scenarios they would use during the visit, which they discussed and role-played, including responses to questions the service provider or other facility staff might ask. The following scenarios were presented during the mystery client visits: Unwanted pregnancy Information regarding contraceptives Information regarding STIs Counseling regarding premarital intercourse Mystery clients were instructed to assess and report on communication skills, waiting time, length of service provided, use of BCC materials, provider attitudes/barriers, cost of service, and their overall satisfaction with the service. Youth were interviewed following their clinic visits using an interview guide (see Appendix B). Results were reviewed by the facilities and partners, and actions were taken to improve any areas that were identified as needing improvement. The following section discusses the various evaluation activities used (facility reassessment, analysis of training data, analysis of client satisfaction data and trend analysis), including methodologies, data limitations, and results. 13

20 Facility Reassessment Evaluation Methodology The primary means of evaluating the facility strengthening activities was through a reassessment of facilities using the facility assessment tool and certification tool to receive an endline score. It should be noted that a subset of questions were selected from the facility assessment tool, specifically those that had data available and were considered the important to achieving YFS. These were: privacy ensured, competent staff, minimum package of services, peer providers/counselors available, and publicity for YFS. Due to limitations in time and resources, a sample of five facilities was selected for reassessments, including both NGO and district partner facilities. The facilities reassessed were: Komamboga Health Center in Kampala Kinyamaseke Health Center in Kasese Bubuto Health Center in Mbale Kasheregenyi Health Center in Kabale Bufulubi Health Center in Mayuge Endline assessments were conducted in January 2005 and the certification tool was applied to those results in February Data Limitations One limitation to the evaluation of the facility assessment and strengthening was that due to funding limitations, a small sample (five out of the 76 facilities) of facilities were reassessed. The results found may not be representative of results across all facilities. Another significant limitation was that in reviewing the baseline assessment results, district staff focused on areas that they believed were most important or on areas which they could have a significant impact. Therefore, each district focused on different areas. However, all assessed the following five out of 14 areas: privacy ensured, competent staff, minimum package of services, peer providers/counselors available, and publicity for YFS. As a result of this, the evaluation only examined the above-listed five areas in the sampled facilities (as opposed to reassessing all elements of the facility assessment tool). Finally, despite requests from program staff for more information on the facilities, including reasons for their improvement in particular areas, this information was not collected and could not be included in this report. Therefore, analysis of the results is limited. 14

21 Results Improvements were seen in each of the five elements, as seen in figure 3. The peer provider/counselors available component had the greatest score increase over baseline, due to the program s peer provider training and program implementation. Another big increase was seen in the privacy assured element. Privacy was an area that was found to be very weak during the baseline assessments and was strongly emphasized during the trainings of service providers and improved through the partitioning work done in many clinics. Figure three provides a comparison of the baseline and endline scores (with 10 being the highest score possible): Figure 3: Total Component Scores Total Component Scores Score Baseline Endline Privacy ensured Competent staff Minimum package of services Peer providers/ counselors available Publicity for YFS Component As can be seen in table 2, each facility also saw overall improvement from baseline to endline; scores for each element either improved or remained the same. Komamboga saw the greatest improvement, with an increase in scores from one at baseline to an endline score of nine. Bufulubi, scoring the lowest of the sampled facilities with a baseline score of zero, made the next greatest gain of the facilities with an endline score of seven. Kinyamaseke, though scoring the highest of the sampled facilities at baseline, saw the smallest gain of three points. The results for each facility are as follows (see Appendix B for complete scores): 15

22 Table 2: Facility Baseline/Endline Scores FACILITY BASELINE SCORE ENDLINE SCORE IMPROVEMENT Bubuto Kinyamaseke Komamboga Kasheregenyi Bufulubi During the assessments, staff noted that one strength of the assessment process was the production of facility-based reports, which allowed for monitoring of the facilities using the specific findings of the assessment. This helped focus the activities on the facilities needs to increase availability of quality YFS services. However, staff also noted that the process of developing action plans in some cases created expectations that were not able to be met through AYA, occasionally causing frustration to some facility staff. 16

23 Analysis of Training Data Evaluation Methodology The end of project evaluation concentrated on the in-service, rather than pre-service training for providers conducted by the national training team. All of these trainings were evaluated using a combination of pre- and post-test assessments and an end of training evaluation. In analyzing the pre- and post-test assessments, average scores gained per training were calculated, and comparisons were made across the different types of trainings. A review of training evaluation summaries and discussions with national and district trainers provided additional information. Data Limitations Training reports mentioned the use of daily reviews and observation to monitor participants learning, but since the trainers did not report the outcomes of either of these, these data were unable to be included in the analysis. Results The following table shows the total participants of both the in- and pre-service work. Table 3: Total Number of Participants Trained Training and Participant Type Number Trained In-service National Trainers 20 District Trainers 28 Service Providers 202 Pre-service Tutors 18 Health workers 48 Preceptors 17 Analysis of pre- and post-tests showed gains in all service provider trainings, as seen in the table below. 17

24 Table 4: Pre- and post-test analysis Service Provider Trainings for 2003 to 2004 District Average Score Pre-test Post-test Gain Kasese Kampala Kabarole, Kasese Kyenjojo, Kamwenge Kampala Kabarole Mbale, Sironko, Soroti, Kaberamaido, Kapchorwa Kabale, Kamwenge, Kyenjojo, Kabarole Mbale, Sironko, Soroti, Kapchorwa Iganga, Mayuge TOTAL The range of score increase for the 10 service provider trainings was from 9 to 30, with an average increase of 20. Those trained felt that they gained valuable knowledge and skills as a result of the AYA/Pathfinder trainings. One national trainer described her observations of how the training transformed the service providers, "At the beginning, providers were asking questions like Why the youth? Are they not served? Where are they? But by the end of the training, they were excited that now they knew what they were going to do to serve young people." A weakness of the training, according to training evaluations, was that many of the service providers would have preferred more time for practical training to further develop their confidence in serving young people in their facilities. 18

25 Analysis of Client Satisfaction Data Evaluation Methodology The primary means of assessing client satisfaction was through analysis of mystery client interview forms from three of the districts that conducted mystery client visits: Mbale, Kasese, and Kabale. In the three districts, a total of 99 mystery client visits were conducted at 22 facilities between April and November The number of visits conducted 8 and the communities within each district in which the visits occurred are included below: Kasese (59): Kiteso, Kahokya, Kuyateka, Malisa, Muhokya, Kinyamaseke, Kitsutsu, and Nyakatunzi Kabale (8): Rubaga, Katenga, Muko, Bufundi, Kasheregyenji, and Kakoomo Mbale (28): Buwundu, Bukokho, Bunatsimi, Bumushikho, Bunamubi, Bumatanda, Bunabutiti and Buchida The table below shows the mystery clients by district, age, and gender. Table 5: Mystery clients by district, age group and sex Age Kabale Kasese Mbale Unknown (years) M F UK M F UK M F UK M F UK Unknown Totals Interview data were received from the partners and entered into Excel for analysis. The data were analyzed by question and are reported below. Data Limitations The primary limitations were the lack of response by mystery clients to a number of questions 9 and the lack of staff time and funding to follow-up more intensely with mystery clients to complete interview forms. Another limitation was the difficulty of measuring quality of care, given the emphasis on mystery clients as the sole source of data. Additional data gathering could assist in determining whether services provided are of acceptable quality. A number of limitations of the interview guide were also identified, including: Questions were not grouped together thematically to enable the clients to reflect and respond in a logical fashion that clearly presents information. The recommendation 8 It is unclear from the forms in which districts four of the visits were conducted. 9 Because of this limitation, the percentages in the graphs do not equal 100 percent; the percentage of nonresponse for each element has not been included. 19

26 would be to group all communication questions together as well as the attitude/barrier questions. The use of a majority of close-ended questions limited the data analysis. For example, it would have been more useful to follow up clients who described services as cheap, affordable, or expensive with a question about how much they paid. Clients who said they would recommend the service to a friend, should have been asked, why? However, even when probing follow up questions were included in the guide, no responses were recorded. It is unclear whether this is because the interviewers did not probe. The rankings and terms used in some questions might also have limited the data. For example, in question 36, cheap, affordable, and free service could mean the same thing to respondents. Finally, the exercise did not address several key issues of youth-friendly services, including confidentiality, hours of operation, and comfortable environment. Results The mystery client visits showed positive results in most of the elements under observation, including the communication skills of the service providers, the waiting and counseling time of the visits, privacy, use of visual aids, cost of service, and overall satisfaction of the visit. The individual results are shown in the following graphs. 20