Document of The World Bank FOR OFFICIAL USE ONLY Report No: 32667 IMPLEMENTATION COMPLETION REPORT (SCL-42070 TF-29455) ON A LOAN IN THE AMOUNT OF US$42.5 MILLION TO THE REPUBLIC OF INDONESIA FOR A SAFE MOTHERHOOD PROJECT June 30, 2005 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective January 1, 2005) Currency Unit = Indonesian Rupiah (Rp) Rp 1 million = US$ 107.20 US$ 1.00 = Rp 9,328 FISCAL YEAR Government of Indonesia: January 1-December 31 ABBREVIATIONS AND ACRONYMS ANC Ante Natal Care BDDs Bidan di Desa or Village Midwife BAPPENAS National Development Planning Agency BEONC Basic Emergency Obstetric and Neonatal Care BKKBN National Family Planning Coordinating Board CAS Country Assistance Strategy CEONC Comprehensive Emergency Obstetric and Neonatal Care CPS Central Project Secretariat DHS Demographic and Health Survey FP Family Planning GOI Government of Indonesia ICR Implementation Completion Report IDP Integrated District Planning IEC Information, Education and Communication KPDT Ministry of Development of Disadvantaged Regions MCH Maternal and Child Health MH Maternal Health MONE Ministry of National Education MOH Ministry of Health MORA Ministry of Religious Affairs MSW Ministry of Social Welfare MTR Midterm Review NGOs Non-Government Organizations PMU Project Management Unit PTT Contract Doctor or Midwife RH Reproductive Health SC Steering Committee SMG Safe Motherhood Grants SPADA Support for Poor in Disadvantaged Area Project TPC Targeted-Performance-based Contracts VSC Voluntary Secure Contraception Vice President: Jemal-ud-din Kassum Country Director Andrew Steer Sector Manager Fadia Saadah Task Team Leader/Task Manager: Puti Marzoeki INDONESIA ID-SAFE MOTHERHOOD CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 4 5. Major Factors Affecting Implementation and Outcome 12 6. Sustainability 13 7. Bank and Borrower Performance 13 8. Lessons Learned 14 9. Partner Comments 15 10. Additional Information 21 Annex 1. Key Performance Indicators/Log Frame Matrix 22 Annex 2. Project Costs and Financing 25 Annex 3. Economic Costs and Benefits 28 Annex 4. Bank Inputs 29 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 31 Annex 6. Ratings of Bank and Borrower Performance 32 Annex 7. List of Supporting Documents 33 Project ID: P036956 Project Name: SAFE MOTHERHOOD Team Leader: Puti Marzoeki TL Unit: EASHD ICR Type: Core ICR Report Date: June 29, 2005 1. Project Data Name: SAFE MOTHERHOOD L/C/TF Number: SCL-42070; TF-29455 Country/Department: INDONESIA Region: East Asia and Pacific Region Sector/subsector: Health (97%); Central government administration (3%) Theme: Population and reproductive health (P); Participation and civic engagement (S) KEY DATES Original Revised/Actual PCD: 09/01/1995 Effective: 11/02/1997 09/23/1997 Appraisal: 04/03/1997 MTR: 06/30/1999 10/02/2000 Approval: 07/01/1997 Closing: 05/31/2003 12/31/2004 Borrower/Implementing Agency: REPUBLIC OF INDONESIA/MOH/BKKBN Other Partners: STAFF Current At Appraisal Vice President: Jemal-ud-din Kassum Jean-Michel Severino Country Director: Andrew Steer Richard Calkins (Acting) Sector Manager: Fadia Saadah Samuel S. Lieberman Team Leader at ICR: Puti Marzoeki Fadia Saadah ICR Primary Author: Teresa Ho 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: M Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: S S Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The Project's development objective (PDO) as stated in the PAD was to assist the Government of Indonesia (GOI) to improve maternal health status in selected districts in two provinces (East and Central Java) by (i) improving demand for and utilization of quality maternal health services; (ii) strengthening the sustainability of maternal health services at the village level; (iii) improving quality of family planning services; and (iv) preparing adolescents to lead a healthy reproductive life. These provinces were selected for their more mature administrative capacities, and were intended to serve as best practice example. To keep the Project manageable, it was decided to limit the Project's geographic coverage to ten districts in the two project provinces. As prominently stated in the Project's name and PDO statement, two special approaches were to be applied: partnership and a family-centered approach. Partnerships were a particularly critical element of Project design. This Project represented an effort to operationalize agreements of the Cairo Conference on Population and Development, which called for family planning (FP) services to be fully integrated into the broader package of safe motherhood services. This called for close partnership between the Family Planning Board (BKKBN), which was responsible for the FP program overall and for demand-generation through promotion and IEC, and the Ministry of Health (MOH), which managed health services, including women's health and FP services and between the public and private sectors (private practitioners, NGOs, communities). The adolescent reproductive health component included three other agencies (education, religious affairs, and social welfare). The family-centered approach involved working with adolescents, male partners, and the communities, with the latter organized to support family empowerment. The Project's objectives were clear, realistic and important for the country/sector and were consistent with the Bank's CAS, which stressed the key themes of human resources development, promoting poverty reduction by increasing access to and quality of basic services and enhancing sustainability. The Project provided an appropriate response to the important issue of high maternal and neonatal mortality. The key project strategies (two-pronged approach promoting demand and supply, piloting innovative/alternative approaches, enhancing private sector participation) was appropriate but it also lead to an unavoidable complex institutional arrangements. However, restricting central level implementation to the two key agencies (BKKBN and MOH) while providing them resources for coordination with the three other central agencies would probably have reduced administrative complexity at minimal cost to the PDO. 3.2 Revised Objective: The Project's objectives were not revised during implementation. 3.3 Original Components: As originally designed, the Project consisted of three distinct set of activities, one at the province level, one at the central level and one to support project administration. The provincial component contained the core activities of the Project, under which essential service delivery operations and demonstration/pilot activities would take place. As described in the Loan Agreement, the Project components were as follows: A. At the province level, the project components included: A.1 Improving the maternal health (MH) status and health services at the village level through (a) - 2 - promotion of maternal health services through IEC-related activities; (b) improving the supply of maternal health; (c) strengthening sustainability of maternal health services through testing alternative service delivery models like the Targeted Performance-Based Contracts (TPC) and research and evaluation relating to the pilots; A.2 Increasing the demand for and access to high quality family planning (FP) services through (a) IEC for FP; (b) training in FP services and counseling skills; (c) provision of equipment and supplies; (d) and quality assurance (QA) programs; A.3 Preparing adolescents for healthy reproductive (ARH) lives by increasing their knowledge of reproductive health issues through (a) assessment of counseling needs; (b) counseling and educations services through parent, family and community education programs and in-school and out-of-school programs; and (c) mass media campaigns. B. At the central level, project components included: B.1 Technical support and training for central and provincial level staff through (a) development of strategies and guidelines for training of health service providers, IEC for maternal and reproductive health, and adolescent reproductive health; (b) training and fellowship for central and provincial staff; and (c) capacity-building at provincial level; B.2 Policy, research and evaluation activities including: (a) policy studies and dissemination of findings; (b) monitoring and evaluation (M&E) activities; and (c) TA to various implementing agencies for M&E activities C. Project administration support included project management units at central and provincial levels. Overall, the components related directly and appropriately to achieving the objectives. Strategic approaches applied in individual activities were generally appropriate. The demand-supply split used in both MH and FP sub-components was an astute way to organize project activities, applying lessons from earlier operations that showed that neither supply-side nor demand-side interventions alone would be effective. The central level components, including project administration, were generally appropriate to the project objectives. The history of "turf battles" between BKKBN and MOH and the limited success of attempts to strengthen coordination between them under several previous Bank-financed projects would have called for alternative arrangements, but there was no practical alternative (except better coordination at the local level) given the key roles of these agencies in the MH and FP programs in the country. 3.4 Revised Components: A number of changes were introduced to the Project during implementation, some in response to two important external factors that affected the project profoundly ­ the economic crisis of the late 1990s and decentralization of 2001. In August 1998, the Loan Agreement was amended to allow for the purchase of contraceptives following the onset of the economic crisis to ensure continuity of contraceptive supply. USD 7.5 million was reallocated to a new disbursement category for contraceptives, later reduced to USD 5.1 million. Project funds reserved for emergency obstetrical services reduced from an initial allocation of USD3.4 million to zero in April 2002 as a very similar approach (to a large extent influenced by project design) was adopted as part of the social safety net program (SSN) initiated in response to the economic crisis in 2000. - 3 - In April 2002, an amendment to the Loan Agreement added a fourth activity to the Province-level component, the Safe Motherhood Grants (SMG). Project funds would thenceforth be made available directly to project districts following an Integrated District Planning (IDP) process. The IDP would incorporate the same range of activities as the original province-level component, adapted to district-level conditions and priorities. In addition to being an appropriate response to the decentralization of health services in 2001, and a reflection of the flexibility maintained by both the Borrower and the Bank during project implementation, the introduction of Safe Motherhood Grants facilitated the integration of MH and FP services and the coordination of demand and supply side activities, and helped achieve partnerships where it mattered most, at the operational level. A number of other changes were also introduced in the project design. These included: l After the mid-term review (MTR), involvement of MONE, MSW and MORA as implementing agencies for the ARH component, was terminated to simplify project implementation. l Although the TPC pilot carried out during the first half of the project was relatively successful, this activity was only implemented until 2003 following a change in policy that permitted local governments to extend contracts of BDDs beyond the initially set maximum of six years. l In 2000, agreement was reached to use loan funds to finance the Indonesian Demographic and Health Survey (IDHS) of 2002 rather than local funds following the crisis. 3.5 Quality at Entry: The Project was reviewed by QAG and its quality at entry was assessed and was rated Satisfactory. The Project had several strong points. Its technical concept was sound and was consistent with CAS and government priorities, and project components were appropriately linked to the PDOs. The emphasis on demand and supply sides was a strong point of project design. A social assessment was prepared and key project interventions such as IEC activities and financing of basic and referral obstetric services were aimed specifically at families at the margin, i.e. families that were poor and/or lived in remote areas. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: As proposed in the project design, achievement of the PDO is not measured in terms of changes in maternal health status or maternal mortality and morbidity but in terms of the four intermediate outcomes. This is an acceptable approach given the well-known difficulty of measuring changes in maternal mortality and morbidity in a small population and over a limited number of years. The data used in assessing Project outcomes have some limitations. As discussed below, the baseline survey was delayed and had limitations in terms of data quality. The DHS was also delayed due to the economic crisis but still provided valuable data to the project that asked to have project districts over-sampled to provide the needed estimates for the project at relatively low cost. Intermediate outcome 1: improving demand for and utilization of quality maternal health services This outcome is rated marginally satisfactory. The main indicator to measure this intermediate outcome is percentage increase in deliveries by trained health providers. Data from the Susenas (National Socio Economic Survey) showed that deliveries by midwives and other trained health providers were increased in all ten project districts (see table below). However, only 4 districts (Cilacap, Pemalang, Trenggalek, and Sampang) achieved the targeted performance of at least 50% increase. One should note that the external - 4 - factors noted above could have had a negative impact on these indicators, hence, the overall positive trend is quite important. On the other hand, the number of cases with obstetric complications referred to health facilities providing emergency obstetric care was very low. Perhaps the original design that called for demand side financing and support from community and village midwives was more appropriate but this is not possible to test since that activity was stopped (as noted earlier -- this was related to the social safety net program). District/City Susenas 1999 Susenas 2004 Central Java Cilacap 36.45 54.82 Rembang 41.65 51.55 Jepara 46.81 57.28 Pemalang 43.33 64.78 Brebes 40.63 48.60 East Java Trenggalek 43.59 69.33 Jombang 72.98 90.80 Ngawi 70.45 84.85 Sampang 11.27 25.73 Pamekasan 25.04 33.50 Beside the main indicators, selected output indicators were also reported six monthly as part of project monitoring. Some of them are as follows: l cesarean section as % of total deliveries at hospitals (CEONCs) declined from 30% in 2002 to 21% in 2004, a reflection of early and better planned deliveries. l % with first Ante-Natal Care (ANC) visit: increased from Dec. 2001 to Dec. 2003 from 84% to 94% but declined to 85% in June 2004; l % with fourth Ante-Natal Care (ANC) visit: similar pattern as with % with first ANC visit. Intermediate outcome 2: strengthening the sustainability of maternal health services at the village level; This outcome is rated marginally satisfactory. The TPC pilot was successfully implemented and useful lessons were learned with respect to the use of demand-side interventions (vouchers). This was a major accomplishment as it is one of the first efforts to demonstrate demand side financing in Indonesia. It should also be noted that the project was meant as a demonstration activity because having a full experimental design was not feasible. It is a pity that the pilots were stopped earlier than planned and not clear if it was possible to continue this effort. The pilots have been used and studied by a number of researchers. For instance, there is a study by Lipiduta Pertiwi done in 2000 that showed that TPC midwives performed better than non-TPC midwives for first ANC visit and skilled deliveries (67.3% and 45.3% respectively above target in TPC areas, 63.7% and 33.7% in non-TPC areas), but there was no observed difference for neonatal visits (92.7% versus 92.8%) and TPCs performed worse for the fourth ANC visit (52.7% versus 60%). The study also showed that 74% of poor women in Central and East Java who received vouchers (in 2000) used them, presumably - 5 - to have skilled deliveries. This compared favorably with data from the Project baseline survey that showed 26.1% of poor women in Central Java had skilled deliveries in December 1999. Another study done later in 2002 showed different results regarding the TPC midwives. According to that report, the TPC midwives did not do better in relation to specific service delivery measures (first ANC, fourth ANC, skilled delivery and so forth). It is not possible to make statistical conclusions regarding this pilot as it was not designed as such (that was not feasible). What is more important is that since that pilot, a number of other programs copied the approach and the concept of using vouchers has gained momentum and acceptability. In fact, a new project led by the Ministry of Development of Disadvantaged Regions (KPDT) is now planning to implement this concept on a large scale and a larger set of activities, building on the lessons learned from this project. Other sectors (e.g. education) are also piloting such approaches. Intermediate outcome 3: improving quality of family planning services This outcome is rated satisfactory based on the observed decline in complications and in drop-outs. l % complication (side effect) decreased from December 2001 to June 2004 from 0.52% to 0.12% l % drop-out decreased from December 2001 to June 2004 from 16% to 5% One should note that according to the PAD, the project was expected to develop a new indicator for quality of family planning. During implementation, that proved to be more difficult and the project then adopted more traditional indicators to measure quality of family planning services. Intermediate outcome 4: preparing adolescents to lead a healthy reproductive life This outcome is rated satisfactory because of the substantial increase in knowledge levels of adolescents (both male and female) and the usefulness of youth information centers as a source of information for youth. l Knowledge indicators - substantial increases in knowledge between 2001 baseline survey and 2003 end-line rapid survey for male and female adolescents on all nine aspects (covering RH, sexuality and HIV/AIDS) l Qualitative studies also show usefulness of youth information centers as a source of information for youth In brief, despite the limitations in evaluation data and methodology, evidence on the four intermediate outcome indicators ­ and therefore for the project as a whole ­ is generally positive. Given at least partial improvement on most measures of utilization and quality of MH services, successful completion of and valuable lessons learned from the TPC pilot on sustainability, improved quality of FP services, and substantial improvements in knowledge levels of male and female adolescents, project outcome is rated satisfactory. 4.2 Outputs by components: A. Province-level Components: A.1 Improving the maternal health (MH) status and health services at the village level A.1(a) Promotion of maternal health services through IEC-related activities It is difficult to judge success of this activity. Outcome measures show no stable pattern - 6 - of differences in knowledge of five danger signs of maternal complications between project and non-project districts in the two project provinces (data source: 2003 IDHS and 10 SMPFA district evaluation), implying that MH-IEC activities were not effective. It is also not possible to determine whether the increase in skilled deliveries over the life of the project could be attributed in any way to these activities. Community dialogue intended to empower families and promote maternal services was introduced later in the project and is perceived by program managers as effective, but there is no data to demonstrate this. Finally, during most of the project lifetime, BKKBN (which was responsible for implementing IEC activities) tended to overemphasize FP over MH. Institutionally, there was a clear division of labor, with MOH targeting its IEC activities to health workers and BKKBN to the general population. However, that is not as easy in practice and it limits the ability to coordinate the messages more closely. This is a recurring concern in Indonesia and although the project design took into account the lessons from an earlier population project that worked with BBKN and MOH, it was still difficult to overcome. By all indications, this activity would appear to be marginally unsatisfactory. A.1(b) Improving the supply of maternal health services As of June 2004, most of the training activities planned for midwives on inter-personal communication and counseling (IPCC) and basic normal delivery had achieved at least 80% of planned coverage. Midwife training in life-saving skills failed to reach 80% coverage in all but 2 of 10 project districts largely because of resistance from obstretrician-gynaecologists to teaching life-saving skills to midwives. This resistance from the specialists is an important obstacle to future improvements in emergency obstetric services and calls for considerable further efforts at dialogue under central MOH leadership. An important contribution of the project is the establishment of DTCs (District Training Center) for basic obstetrics training, and implementation of peer reviews to monitor performance post-training. The peer review was financed by a revolving fund managed by the midwife association, and will hopefully be continued post-project. Another helpful tool for improving service quality, particularly for management of emergency cases, was the maternal perinatal audit (MPA) involving all midwives in a particular district and facilitated by the obstretrician-gynaecologist from the hospital. The sessions identified and analyzed causes of death in a pre-selected maternal death cases and were meant as a learning forum for the participants. Other activities planned under this component included the following: l standard procedures for emergency obstetric and neonatal care ­ the Project disseminated the standard operating procedures developed under the USAID-financed Maternal Neonatal Health (MNH) Project l development of guidelines for anemia control ­ guidelines were developed for supplementation for all women of reproductive age, not just pregnant women; partnerships were established with producers of iron tablets to conduct IEC, ensure supply and develop outlets for iron tablets. l provision of equipment to maternal health facilities ­ procurement of midwifery kits, - 7 - incubators and other equipment for BEONC were included in SMGs l carrying out of surveys and research on maternal health issues at provincial level ­ most surveys and research were conducted by the central level, only a few were done by the provinces l provision of emergency obstetric care at public and private facilities to needy patients ­ funding for this was dropped after introduction of the social safety net program in 2000. Given the relative success of training activities, the establishment of District Training Centers, the peer review process, the use of maternal perinatal audits, and generally successful implementation of other activities implemented under this component, this component is rated satisfactory. A.1(c) Strengthening sustainability of maternal health services through establishment of pilot group practice models, a pilot scheme of Targeted Performance-Based Contracts (TPC), refurbishing and equipping of private practice clinics for BDDs, and research and evaluation relating to the pilots; This activity was introduced as a response to a MOH policy existing at the time of preparation that limited BDD contracts to a maximum of two 3-year terms each. The idea was to allow BDDs reaching the end of their terms as public contractors to convert into successful private practitioners who would survive financially through 3 sources of income ­ a basic salary, claims for services provided to the poor (with performance-based incentives operating through a voucher system), and fee-for-service from non-poor families. Two types of pilots were planned: (i) group midwifery practice models and (ii) targeted performance-based contracts. Though started with some delay (partly because unstable prices during the economic crisis caused difficulties in setting appropriate remuneration levels), the TPC pilot was successfully implemented during the first half of the project. However, two changes in national policy in 2002 ­ permitting an unlimited number of contract renewals for BDDs and introducing the SSN program that distributed payments directly to the health center limited the scope for expanding this pilot further. Efforts were made to start some other pilots but in light of the economic crisis and the introduction of the SSN (that built to a large extent on some of the features of this project), it was decided to focus the project implementation and be more selective in the number of pilots to be initiated at the time. Useful lessons were learned from the TPC pilot about the feasibility of using vouchers as a performance-based payment mechanism, which lessons are being picked up in other projects outside the health sector today. This activity is rated satisfactory. A.2 Increasing the demand for and access to high quality family planning (FP) services Training in FP services and counseling was provided for midwives, doctors, and FP field workers. Training for FP workers in counseling was the most successful in terms of coverage and, as claimed by BKKBN management, in terms of impact on service quality. The project did not reach target coverage for training on clinical FP standards and on FP management, largely because project managers were hesitant about using project funds for training of private sector providers and there was little interest among doctors in receiving training from BKKBN. The voluntary surgical sterilization (VSC) pilots were started during the project extension period. Training in VSC was completed as planned for FP field workers and midwives. Pilot progress - 8 - were also implemented including the introduction of problem based learning for family planning and reproductive health services in midwivery schools, and a revolving fund model for private practicing midwives to improve availability of contraceptives at the village level. Strong advocacy efforts at the local government are needed to sustain these pilots Two approaches to quality monitoring of FP were developed under the project ­ one by BKKBN, for program management quality; a second one by MOH, for technical quality. Various tools and approaches on FP quality applied were adopted and expanded from the USAID-funded STARH. The existence of several tools for the FP services caused confusion at field level. Still some project districts have chosen to allocate funds for QA teams after the project, a positive signal of the system's sustainability post-project. Because targets for FP training were only partially achieved, and because of the multiplicity of tools linked to QA activities this component is rated marginally unsatisfactory. A.3 Preparing adolescents for healthy reproductive (ARH) lives by increasing their knowledge of reproductive health issues The operational aim of this component was to test different approaches to engaging adolescents in RH activities. Prior to the MTR, five government agencies were involved in implementing the component, with each Ministry testing different vehicles for reaching the target group. Project activities were slow to start under this component and, after MTR, a change in implementation arrangements was introduced, accompanied by a change in strategy for reaching adolescents, with the project shifting emphasis to district-run youth information centers, providing outreach services centered on peer-to-peer counseling. Given overlap in target groups of MORA and MSW with those of BKKBN and MONE, the former two were dropped as implementing agencies. MONE was also dropped, even though schools remained an important channel for achieving ARH objectives, because they were slow to formulate their interventions. An NGO was engaged instead to develop the new BKKBN-led design, centered on the peer counselor and peer educator approach, to more effective ends. Although involvement of central-level MORA, MONE, and MSW was terminated, their district level offices (Dinas) were actively involved in implementation of the SMG. Some local governments have chosen to continue supporting youth centers through their budget after the project closed. School-based interventions continued through the post-MTR period. Two target groups were identified: · children 15-19 years old were originally to be reached through inclusion of ARH topics into the curriculum. MONE carried the program as an extra-curricular activity. No evaluation has been done of the program, but it is reported to be popular. · children 10-14 years old were added on the premise that reaching children in their early adolescence would have a lifetime impact on their behavior. Although this approach is reported to be working, judging from the good response from parents and students, there is no evaluation of its effectiveness. Nevertheless, the program is now adopted into national policy MOH's ARH-related activities started initially as a program to train nurses and doctors on ARH. - 9 - In 2003, after an evaluation showed low utilization of the services because of a lack of demand, MOH initiated a pilot on youth-friendly health services implemented in schools through school health programs and with NGOs and youth groups to reach out-of-school youth. There is no information on the outcome of this pilot. MOH should consider working more closely with others instead of insisting on their other programs. This component is rated satisfactory especially given the difficulties many countries face in starting such programs. A.4 Safe Motherhood Grants (SMG) to selected Kabupaten Governments The SMG component was the most successful innovation of the project, and is credited for the turnaround in implementation. Its success can be attributed to several factors: the application of the integrated planning process/tool (the Integrated District Plan or IDP) as the basis for fund allocation and implementation, increased involvement of district level officials, improved multi-sectoral coordination among implementing agencies at the district level, the transfer of project funds through a block grant/matching grant approach (the SMG) directly to district authorities on the basis of the approved IDPs, and the support provided by central level agencies during preparation and implementation of IDPs. The main issue encountered in implementation of the IDP/SMG was its high cost of preparation and administration. Preparation costs will likely be reduced for any future replication, given the steep learning curve in developing the IDP process. Implementation, on the other hand, involved the appointment of a full-time District Facilitator responsible for multi-sectoral coordination and, essentially, to ensure (facilitate) implementation, a position that was critical to success. Most importantly, this change fitted well with the change in government policy and the new laws on decentralization. It also fitted well with other decentralized health programs that the government and the World Bank were supporting. Overall, the high cost administration was minor compared to the favorable outcomes generated by this component. For demonstrating the potential advantages of decentralization so soon after it was introduced as national policy and for substantially improving implementation of province level activities, this component is rated highly satisfactory. B. Central-level components: B.1 Technical support and training for central and provincial level staff BKKBN and MOH prepared strategies and guidelines for training of health service providers, IEC for MH and FP, and ARH but several cases of conflicting guidelines and duplication, especially in QA for FP, MH-IEC and potentially for outreach activities for ARH reflect the lack of coordination. MONE, MSW, MORA also contributed to preparation of guidelines for ARH but were dropped as implementation agencies after MTR to simplify management structure. Instead, they continued involvement in planning and implementation of the SMGs through their local district offices. This component is rated marginally satisfactory. B.2 Policy, research and evaluation activities including: (a) policy studies; (b) monitoring and evaluation (M&E) activities; and (c) TA to various implementing agencies for M&E activities - 10 - There were delays in both monitoring and evaluation activities. The baseline survey for the MH and FP components was not of adequate quality and the team deviated from the initial intent to just build on the Demographic and Health Survey (DHS). The DHS, planned to be completed in 2001, eventually took place in 2002/03. In addition to M&E activities, numerous studies were planned under this component but there were difficulties in quality of initial proposals. Still, a number of useful innovations/pilots/studies were completed, including the TPC, ARH, VSC centers of excellence, problem-based learning approaches for midwives, marketing public relations, iron tablet distribution, special community diagnosis and special interventions for Madurese (a difficult population to reach and support). Small-scale evaluations of some pilots were done. Despite the difficulties and some delays, a number of the studies/pilots had direct influence and have been adopted into national policy or financed by local government after the project. It is difficult to rate this component. On the one hand, it generated a large number of useful pilots and innovations. On the other, monitoring and evaluation activities did not deliver the desired outputs. The component is rated marginally satisfactory. C. Project administration support including project management units at central and provincial levels. As already discussed, project implementation arrangements were complex, particularly at the central level. Following the Cairo conference, this project was the first attempt to clarify the roles of BKKBN and MOH in implementing the concept of Safe Motherhood. The solution proposed - assignment of demand-side activities to BKKBN and supply-side to MOH ­ was only partially successful. There were gaps on the demand side (MH-IEC initially) and overlaps on both demand and supply sides (MH-IEC later, FP and MH-QA, ARH, VSC services, which should have been provided by MOH, not BKKBN). Still, the initial complex arrangements at the central level eventually improved with the dropping of MONE, MORA and MSW as implementing agencies after the MTR. Also, the shift in project management strategy to decentralized planning and implementation diminished the scale of the problems at the central level. Because of the delay in decentralization of the FP program, most IDPs tended to give smaller allocations to BKKBN's FP activities. With most districts having completed their reorganizations by the time FP became an obligatory function for the district, BKKBN and its network of FP workers became essentially homeless. In many cases, their functions were merged with those of either demography or statistics units. Improvements of project management at the central level continued to take place during the life of the project. The CPS at BBKN was strengthened over time, especially after it moved out of the planning to one of the implementing units. A focal point was also appointed at the MOH, improving communication between BKKBN and MOH. Overall, the component is rated marginally satisfactory. 4.3 Net Present Value/Economic rate of return: Not Applicable. - 11 - 4.4 Financial rate of return: Not Applicable 4.5 Institutional development impact: The coordination structure between BKKBN and MOH requires continuing effort ­ through a clarification of each agency's roles, better collaboration on supply and demand side issues, consistency in messages delivered to the general population and to their service providers, etc. ­ if duplication and confusion are to be eliminated. In the decentralized context, in particular, a failure to achieve effective partnership would only lead to both agencies becoming increasingly irrelevant over time, as provinces and districts take the lead. The integrated planning and implementation model at district level has been the Project's most important contribution to institutional development, with likely impact not only on the health sector but on other sectors affected by the government-wide decentralization. It is a good indication of the best case scenario that could result from decentralization, if done properly. The voucher mechanism tested under the TPC pilot also added value beyond the Safe Motherhood Program, demonstrating the logistical feasibility of applying demand-side incentive payment mechanisms using government funding. Renewed effort to reduce their administrative workload and costs would improve the chances that this approach would be picked up more widely by cost-conscious local governments. Other pilot and demonstration activities of the Project yielded useful lessons for effective service delivery ­ e.g., the youth information centers for ARH; the VSC centers of excellence; and cooperation with manufacturers on iron tablet distribution. Even the less successful attempts such as the health facility-based youth-friendly ARH services taught valuable lessons as to which models are not likely to work. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: The 1997/98 economic crisis was a major factor for the Project, causing major delays in implementation and leading to changes in design and implementing agencies' roles. During the crisis, GOI resources to support project activities were severely limited. There was not only a shortage of funds but also of even scarcer managerial resources, as the GOI was understandably distracted by the need to cope with emergencies created by the crisis. The crisis and its aftermath also affected directly certain project activities ­ specifically the BDD pilots and the financing of emergency obstetrical services ­ as discussed above. 5.2 Factors generally subject to government control: There was considerable uncertainty in connection with the Government-wide decentralization implemented in 2001, including with respect to the changing roles of Project's central level lead and implementing agencies. This process was complicated even further by the fact that MOH decentralization occurred largely in 2001 while BKKBN was not decentralized until 2004. Nevertheless, a strategic coup was accomplished when the MTR team (in October 2000) capitalized on the anticipated decentralization to push for the IDP approach. This decision not only ensured that the project (especially the project districts) would be ready for decentralization when it was launched the following year but also finally brought to the fore the needed focus on integration of the safe motherhood program delivery system. 5.3 Factors generally subject to implementing agency control: Difficulties in coordination between BKKBN and MOH were a serious factor, as was the problem with - 12 - staff quality in the project management structure during the first two years of the project. There was not enough leadership support for the program after it started implementation. On a different plane, the resistance of obstetrician-gynaecologists to increasing the level of responsibility of midwives in provision of emergency obstetric services, as well as their general lack of interest in the training and QA activities of the project, indicate that the MOH and BKKBN face an uphill battle in dealing with this critical stakeholder group. 5.4 Costs and financing: The total loan amount was reduced from US$42.5 million to US33.35 million through a series of four loan amendments. The last amendment, in December 2002, also extended project closing by nineteen months. Estimated actual disbursements from the loan totaled US$32.5 million, or 76.5% of the original loan amount. Estimated total project costs (including IBRD and GOI funding) totaled US$35.7 million, or 58% of the original estimated project cost. The main reason for the reduction in project cost was the change in the exchange rate, which increased from Rp 2,348 per USD1 at appraisal to around Rp. 9,300 per USD1 at project closing. 6. Sustainability 6.1 Rationale for sustainability rating: It is difficult to judge sustainability of project outcomes at the district level largely because, with the decentralized implementation arrangements for the Safe Motherhood Program, follow-up activities can be expected to vary from one district to another. Nevertheless, there are strong signs that much of the project's innovations will be maintained in the project districts. All the districts have issued a policy to continue the partnership and family model to improve the safe motherhood post-project. In addition, the East Java government issued a decree to develop a provincial RH committee and requesting all district to develop similar district committees. Also, each district has developed a strategic plan for continuation of the various project initiatives, which plans include budget proposals for selected project activities for 2005 and plans for continuing allocations up to 2009. These district plans are presented in the document Safe Motherhood: Partnership and Family Approach - Experiences from SMPFA Project which was jointly prepared by BKKBN, MOH and the World Bank. In parallel with preparation of these plans, districts implemented an advocacy strategy to legislative and other stakeholders to promote the recommended budgets. As of the time of the ICR mission in May/June 2005, reports were coming in that project districts have allocated significant amounts of resources this year for MH. A number of the project innovations are also being picked up in areas outside the project districts. The voucher system that was applied in the TPC pilot will be applied in seven Provinces through the proposed WB-financed Support for Poor in Disadvantaged Areas Project (SPADA) and vouchers will be adopted by a USAID-funded project for Aceh reconstruction. There is also some discussion in MOH of the possibility of using the TPC payment scheme for government midwives. The FP training packages developed under the project have been expanded to non-project areas. And finally, UNFPA will adopt an SMG-type block grant approach for its upcoming Program. 6.2 Transition arrangement to regular operations: (see the above statements regarding district plans for post-project continuity) 7. Bank and Borrower Performance Bank 7.1 Lending: Bank performance in identification, preparation and appraisal is rated satisfactory. Technical inputs were - 13 - of high quality and project design is considered sound. Complexity was addressed by limiting overall scope and later on by dropping activities that did not perform well, as was initially planned. Although the project built on lessons learned from earlier effort regarding coordination with BBKN and MOH, that was not enough to address the disincentives in the system for the two agencies to cooperate at the national level. Improvements were noted at the provincial and district levels. 7.2 Supervision: Bank supervision is rated satisfactory. The factors causing delays during the first half of the project were identified early and repeatedly during successive supervision aide memoires and in management letters to Government. Unfortunately, Government was slow to respond to these reminders and cautions. The MTR mission's proposal to shift gears to the IDP/SMG approach in anticipation of decentralization and the assistance provided thereafter by the Bank in working out the planning process and flow of funds mechanism, was an important contribution to the project, and merits special mention. The Bank continued to give good technical support through out the project life. 7.3 Overall Bank performance: Overall Bank performance is rated satisfactory. Borrower 7.4 Preparation: Government performance during preparation is rated satisfactory. The Government played a major role in the Project's preparation. There was strong commitment from Bappenas, BKKBN, and MOH to the project. Each participating Ministry prepared its proposal for the project. The provinces were active participants in project preparation workshops and also prepared their own project proposals. Still turf issues persisted and despite assurances provided, those continued to affect the project. 7.5 Government implementation performance: Government implementation performance is rated marginally unsatisfactory. In fact, performance was mixed. The late availability of Government counterpart funds delayed some Project activities, continuing even into the post-crisis period. Bappenas was expected to chair the Project Steering Committee at the central level, but that function did not materialize. On the other hand, active participation by Bappedas at the district level and multi-sectoral cooperation at this level was critical to the success of the IDP/SMG component. 7.6 Implementing Agency: The implementing agencies' performance is rated satisfactory, although actual performance was mixed. Poor performance during the first half of the project can be attributed in large part to the instability generated by the economic crisis, but also to the failure of Government to act on strengthening the project management structures, as well as to undertake other actions to speed up implementation, as repeatedly suggested by Bank supervision missions. Nevertheless, the implementation, even with delay, of the recommended changes, improved implementation and helped achieve positive outcomes, especially given the macro changes they had to deal with. 7.7 Overall Borrower performance: Overall Borrower performance is rated satisfactory. 8. Lessons Learned Several valuable lessons can be learned. These are: 1. Decentralization can work with (a) clear strategies and technical solutions in place; (b) well - 14 - developed tools to aid district-level planning and a district led planning process, and (c) integrated planning and funding mechanisms. Decentralization also helped with multi-sectoral coordination. However, it is important to ensure that districts, at least initially, have access to technical guidance and support. 2. Institutional incentives are key and coordination issues between BKKBN and MOH need further resolution. In the Indonesian context, long-term success of the integrated safe motherhood program will require a clear resolution of the issues of cooperation between BKKBN and MOH. Coordination meetings and workshops, even planning mechanisms may not be enough if stronger incentives for coordination are not in place. Moreover, coordination at the local level is simpler and most likely to succeed as noted earlier. 3. The combination of demand and supply approaches can lead to positive outcomes. The project's emphasis on ensuring that both demand- and supply-side interventions were included probably helped ensure better outcomes. 4. Using vouchers as a tool for demand-side financing is logistically feasible and would be an effective way to target the poor. Despite the limited scope of the pilots, they clearly demonstrated the possible use of vouchers and such demand financing tools to help improve access of services, especially for the poor. This model can also be adopted to the public sector as well as the private sector and can cover a wide range of services. The challenge will be how to coordinate supply and demand side subsidies. 5. The role of medical providers other than doctors is still constrained by the dominance of the medical community, especially specialists. In this case, more work is needed to persuade obstetrician-gynecologists to agree to the expanded role for midwives in managing emergency obstetrical services and to participate in other Safe Motherhood Program activities. 6. Fit the project design to implementation capacity. This can be done by undertaking more extensive upstream work to (a) determine with greater confidence which "essential" parts of the package may be dropped from the project; and (b) for demonstration activities, pre-screen options to be tested and limit these to the most promising ones. 7. Supervision is important and task teams are encouraged to adjust projects depending on the changing circumstances. In this case, the economic crisis, the decentralization and the changing role of the implementing agency were major factors that affected the project. The government and Bank teams worked closely and hard to review and revise the project's scope as needed to keep it relevant and working and that had made a significant difference in its outcomes. 9. Partner Comments (a) Borrower/implementing agency: The original agreement between GOI and the World Bank was to implement SMPFA from August 1997 to May 2003. Later, it was agreed to extend the project to December 2004. The project was implemented in ten (10) districts in two provinces, Central Java and East Java, i.e., Brebes, Cilacap, Pemalang, Jepara, Rembang, Ngawi, Trenggalek, Jombang, Sampang, and Pamekasan. Some project interventions were also implemented at the provincial and central levels. The project adopted a specific approach, i.e., "family and partnership" approach, which was considered unique if compared to other population and health projects in Indonesia. The family approach recognized that improving maternal health required family involvement, as family is the smallest unit of the society. Partnership approach is indicated by the integration of five government sectors and involvement of many NGOs. The partnership includes Ministry of Health (MOH), BKKBN, Ministry of National Education (MONE), Ministry of Religious Affairs (MORA) and Ministry of Social Welfare (MSW), and NGOs, such as Muslimat, Muhammadiyah, IPPF and Family Welfare Club (PKK). The project consisted of 3 (three) main components: the Maternal Health, Family Planning, and Adolescent - 15 - Reproductive Health. The strategy for the Maternal Health component included demand, supply, and systems improvement. Improvement of demand was aimed at improving knowledge, attitude, and behavior on various issues related to maternal and reproductive health at individual, family, and society levels. Supply improvement was aimed at improving the quality of maternal health services, especially in rural areas, improving the quality of emergency services, and improving the referral system. Systems improvement was to guarantee sustainability of the outputs. In the Family Planning component, supply side interventions were to improve the technical skills of family planning personnel and their competence as regards interpersonal relationship and counseling. Demand interventions were to improve knowledge, awareness, and behavior of family planning clients about good quality services. Systems interventions were to ensure continuous delivery management of good quality services. The objective of the Adolescent Reproductive Health component was to improve the knowledge of the adolescents on various reproductive health issues. Supply interventions were to improve accessibility of the adolescents to information, counseling, and medical services on adolescent reproductive health. Demand interventions were to improve the coverage of the adolescents and parents who received information and counseling on adolescent reproductive health, while systems interventions were to improve commitment of the stakeholders on the Adolescent Reproductive Health program. The project was funded by the Government and the World Bank, and the cost sharing between the Bank and GOI was 90:10. The original loan amount was USD 42.5 million. Four amendments were done during the project implementation period. The first amendment was done on August, 1998 brought about by the change in the exchange rate. The loan size was amended to USD 36.5 million. On April 2000, another USD 2 million was canceled in response to the new national policy on foreign loans. The third amendment was done on September 2001, while the fourth or last amendment took place on December 2002 as a part of loan extension. The final loan size was USD 33.3 million, and by the loan closing date disbursement was USD 32.4 million or 97 percent from the amended loan amount. Expenditure by project categories showed that the total loan disbursed directly to the district level through the District Block Grant (SMG) was quite significant, or 24.8 percent. Other expenditure by project categories were IEC (15.9 percent), training/workshop (16.6 percent), and study (11.9 percent). Expenditure for contraception procurement during the economic crisis was 24.8 percent. The financial report showed that expenditure by project components was quite balanced. At the district level, disbursement of the SMGs were around 14 percent for Maternal Health Demand (MHD), 24 percent for Maternal Health Supply (MHS), 24 percent for Family Planning (FP), 19 percent for Adolescent Reproductive Health (ARH) and 19 percent for Project Administration (PA). While at the provincial level, MHD was 14 percent, MHS was 28 percent, FP was 26 percent, ARH was 13 percent, and PA was 19 percent. Project Implementation Although loan agreement was signed in 1997, acceleration of implementation at the field level was only started in 2002. In the beginning, the project faced many internal as well as external problems. Until year 2000, the disbursement level was only around 24 percent. Externally, the economic and political crises influenced project performance, while internally, sectoral and centralized approaches affected project performance. Lack of preparation before project signing for example, delayed establishment of project management including the secretariat team, the absence of guidelines/manuals, the lack of socialization of the project to the relevant stakeholders, and unavailability of baseline data; contributed to low project achievement at the beginning of the project. Administrative issues such as the amendment of the loan amount took a lot of time of project stakeholders and project - 16 - staff. Communication between the project secretariat and the World Bank Office in Jakarta before the midterm review (MTR) was not smooth. The process to get the Bank's No Objection Letter (NOL) was very time consuming. On the other hand, the government administrative process was an additional burden to staff. Some of the problems were due to the low capacity of the human resources. Following the MTR that was conducted in 2001, GOI and the World Bank adjusted the project strategy to respond to decentralization. The recommendation of the MTR was to focus project implementation at the district level through the Integrated District Planning (IDP) beginning in year 2002. The approach was to use integrated planning to meet the demand of the local community. The approach put the District Development Plan Agency (Bappeda) as the project coordinator. The IDP strategy proved to be useful for district policy makers in learning about the Sector Wide Approach (SWAp) for implementing the Safe Motherhood Program. Some characteristics were as follows: 1. The function of the central and provincial level was more in providing technical assistance to the district level. Under decentralization, the roles of the central and provincial offices focused on standardization, developing material prototypes, providing technical support, and overall monitoring and evaluation. 2. Resources for district level activities were allocated under a specific category in the loan agreement and channeled directly to the district level through block grants. This strategy has speeded up project disbursement. Later on, in order to increase local government ownership and the sustainability of some project initiatives, the block grants were combined with matching funds. The block grant strategy is now adopted by Bappenas in other lending program for health. The combination of block grants and matching funds is also proposed by the GOI to UNFPA for the next Seven Country Program 2006-2010. 3. Project secretariats at the central, provincial as well as the district levels were strengthened by recruiting consultants at each level. Meanwhile, the World Bank also assigned a task manager at the Jakarta office. This move was believed to have a positive impact on project performance. The recommendations of the MTR had a positive impact on project performance. Many strategic activities such as project monitoring and project IEC, IDHS and mass campaign were put in place during the MTR. Resources and activities were allocated mainly at the district level. The role of the central and provincial level was more to provide technical assistance to the districts. Disbursement level increased substantially to around 70 percent in early 2003, although implementation of several activities, mostly related to research, was still experiencing delays. Coordination between GOI, i.e., the Central Project Management Unit and the World Bank Office in Jakarta after the MTR was good. Periodic joint planning and monitoring or review missions were very useful for early warning of potential problems and weaknesses in planning. GOI admitted the leadership and capability of World Bank Task Manager in Jakarta in managing the project. Another characteristic of SMPFA project after the MTR was the strong collaboration with other international donors which were also working on reproductive health and maternal health programs in Indonesia. The international donors were USAID through STARH and MNH programs, UNFPA, WHO and UNICEF. Some initiatives and results from SMPFA project were adopted by the three donors' programs and vise versa. STARH program and SMPFA project collaborated closely in efforts to increase the quality of family planning, e.g., in: establishing and empowering centers of excellence in long term method of contraception, developing Interpersonal Communication and Counseling (IPCC), developing and piloting Decision Making Tools (DMT) Manual in family planning and reproductive health (this manual was adopted as the WHO manual), and also in implementing the Integrated District Planning (IDP) model in family planning and reproductive health in STARH program area. Collaboration with UNFPA and UNICEF was mainly on ARH component, in which some ARH manuals and IEC developed in the SMPFA project were reproduced and - 17 - implemented in the two donors' project areas. On the other hand, some modules and strategies developed by UNFPA were implemented in SMPFA project area. During the extension period, the project was focused on completing the planned activities proposed in the original agreement, strengthening strategic activities being implemented, determining strategies for improving mother and child health through SMPFA approach in a decentralized framework, and assisting districts in planning and implementing SMPFA approach. Lessons Learned The interventions or programs implemented in the project were divided into two categories: essential programs and model development. Essential programs had to be done at all project locations on a large scale, and they supported the national policies. Model developments were only implemented in certain project locations as trials. In the second half of project implementation, essential programs in the Maternal and Neonatal Health component were directed entirely to support the national policy of Making Pregnancy Safer (MPS) to accelerate the reduction of Maternal Mortality Ratio (MMR), by building a sustainable and comprehensive Maternal and Neonatal health service system involving both the supply and the demand side. The policy was supported by the project through various efforts including: (1) to improve the quality of basic maternal and neonatal health services at the village level through the Village Midwives (Bidan di Desa) supported by the Traditional Birth Attendants (TBA) or currently referred to as the "Baby Nursing Attendants" who will assist the village midwives in giving post-partum and neonatal care, (2) to improve and increase the quality of emergency maternal and neonatal interventions and referral system at the district level, by improving personnel and institution competencies on maternal and neonatal health referral system, improving the ability of BEONC (PONED) health centers and health centers with beds, and by improving personnel and institutional competencies to implement CEONC (PONEK) at the district hospitals as the highest referral institution in the district level, and (3) to increase and strengthen demand of targeted communities, particularly the poor, in using improved and better distributed maternal and neonatal health services in project areas through demand creation activities, such as the conventional IEC and IPCC approaches implemented by both the health sector and BKKBN, and the Public Relation approach designed specifically to support a marketing program to achieve maternal and neonatal health objectives in the Safe Motherhood program. The essential programs of family planning component included (1) increasing public demand for quality family planning services through IEC and IPCC activities; (2) improving Family Planning providers through training as well as monitoring and supervision activities; (3) monitoring service quality through routine monitoring, post marketing surveillance, and special activities such as implant tracking and removal; (4) improving access to Family Planning services by providing the essential medical supplies, especially Implant Removal Kits, providing contraception for poor families during the economic crisis, improving the quality of storage of contraceptive supplies, and developing the distribution mechanism of contraceptive supplies; and (5) reinforcing Family Planning management by developing a referral system and the Performance Improvement Process (PIP) system of Family Planning & Reproductive Health. The essential programs of Adolescent Reproductive Health component included: (1) improving commitment/support from various parties involved in and responsible for the ARH program; (2) disseminating ARH information through IEC activities; (3) forming and developing the ARH information and consultation centers combined with the empowerment of peer educators and peer counselors; and (4) establishing referral centers to meet the adolescents' need for medical services in reproductive health. Besides the above essential programs, the project also developed several trials of models as pilots in a number of locations. The model trials included the following: (1) pre service training to improve the quality - 18 - of technical skills as well as provider-client interaction in family planning and reproductive health services, among midwifery school graduates; (2) developing the center of excellence for long term contraception methods, involving training, information system, and services, to function as the highest referral center for long term contraception method in East and Central Java; (3) developing a revolving fund model among midwives involved in delivering Family Planning services in order to increase access to family planning and reproductive health by increasing availability of contraceptive at the community level ; (4) implementing the Mother Friendly Movement (GSI/Gerakan Sayang Ibu) to improve husband participation in family planning and reproductive health program; and (5) developing an integrated Safe Motherhood Program at the district level. Towards the end of the project, many investments were done on safe motherhood in the project areas. For the family planning component, the means and infrastructure to store contraception instruments and drugs were renovated to meet existing standards. The means of service were also improved by the provision of sea transportation in Cilacap and Jepara districts. In order to improve the coverage and quality of long term Family Planning method, four hospitals were developed as centers of excellence. Improvement of coverage and quality of services included the improvement of the equipment and infrastructure of the concerned hospitals. The project has also made an attempt to improve the quality of Family Planning education in midwifery schools. This effort was done by applying a Problem Based Learning (PBL) curriculum in 10 midwifery schools. Besides, the project also improved the means of education by providing selected equipment to conduct practice for the midwifery schools. The project also invested in the development of a revolving fund among the midwives who provided Family Planning and Reproductive Health services in Brebes, Jepara, Ngawi, and Sampang districts. The revolving fund was used to purchase contraception instruments and drugs. In the Adolescent Reproductive Health component, the project developed public health centers as adolescents service delivery centers. The project also developed the Adolescents Information and Consultation Center (PIK-KRR) in schools as well as social groups. The PIK-KRR did not provide medical services to the adolescents. However, the sustainability of the public health centers as service delivery centers for the adolescents was uncertain. Three kinds of skills and competencies were believed to have high leverage for the achievement of project objectives. These involved the training of midwives and other health personnel in the health services system, i.e., the midwives, sub health center, health center, and hospital personnel; the Family Planning field workers, and other Family Planning cadres; as well as the facilitators, peer educators and counselors who perform the ARH program. The skills and competencies were: 1. the skills and competencies related to health services to pregnant and delivering mothers, Family Planning and Adolescents Reproductive Health, 2. the skills and competencies related to interpersonal communication and counseling of health and family planning personnel which were important for expanding the coverage of the public as the users of maternal health services, family planning, and adolescent reproductive health and, 3. the skills and competencies in management of maternal health and family planning services, particularly related to quality of services. All districts have invested in improving the quality of human resources in technical as well as interpersonal communication and counseling (IPCC) skills. Almost all health providers and family planning field workers received IPCC training. Unfortunately, the percentage of health providers who received technical training in contraception was still quite low. There were significant differences among the districts in their investment on the improvement of Family Planning technical and management skills. In the ARH component, the - 19 - project put extensive investment on peer educator and peer counselor training activities. The project also developed several systems expected to improve the quality of family planning services. They included the following: l Monitoring system of service quality. The system was carried out in all project districts by the Quality Assurance Team (TJM/Tim Jaminan Mutu) for Family Planning & Adolescents Health services formed under the project. The team established at the provincial as well as the district levels monitored service quality at service delivery points. Monitoring by the team was inseparable from other Family Planning quality monitoring mechanisms, such as after marketing surveillance and internal audit by the concerned service delivery points. l Management of Family Planning by developing a referral system and the Family Planning and Reproductive Health Performance Improvement (PKP KB-KR) model in all project districts. l Referral system on information, counseling, and medical services for adolescent reproductive health. There was a good relationship among the peer educators and counselors and the PIK-KRR (Adolescent Information and Consultation Center) organizers, the professionals, and Adolescent Health Services in all project locations. The Regional Governments had high commitment for the continuation of the ARH program post project. l The integrated planning system of the safe motherhood program at the district level. The system was started by the project in all districts and would be continued post project. All districts of the SMPFA project locations have issued a policy to continue the safe motherhood and reproductive health program after the end of the project. The policy is stated in a number of official documents such as the Regional Regulations and the Regent Decree. The policy is followed by the budget allocation for the safe motherhood program through the Local Government Budget. The SMPFA project approach has been adopted by all districts as an approach that shall be sustained after the end of the project. This demonstrates that the project has increased the commitment of local government on safe motherhood and reproductive health program. The impact of the project on safe motherhood in project areas in particular and national in general should not be evaluated only by outcome or impact indicators indicated in the Project Appraisal Document (PAD). Evaluation based on impact or outcome indicators only, would give misleading conclusion as the intervention period was very short, and as mentioned earlier, acceleration of activities at the field level only started in 2002. Furthermore, most components except for the Adolescent Reproductive Health component did not have an appropriate baseline data. Evaluation of input, process and output indicators would be more appropriate in this case. As demonstrated by the final evaluation consultant, the compliance of informed choice in family planning program by providers in project areas was significantly higher compared to that in non project areas. This indicated somewhat an improvement of quality of family planning services. In the adolescent reproductive health component, the baseline and postline surveys suggested knowledge improvement of adolescents on Reproductive Health issues in project areas. Although the evaluation results of Maternal and Neonatal Health were incomplete, they would still be useful combined with results from other findings. They provided information on project implementation process as well as lessons learned in developing a Safe Motherhood-Making Pregnancy Safer program to accelerate the reduction of MMR. In order to avoid biased conclusions, incomplete data and information gained from the evaluation process would be combined with data and information from other Maternal and Neonatal health reports from the technical units. Even though the ideal condition was not yet achieved, the GOI believed that some project initiatives contributed to the improvement of safe motherhood in project areas. Furthermore, many initiatives and - 20 - models developed in the project have contributed to national policy development and some of the models have been replicated outside project areas. In the family planning component, based on lessons learned from the project, voluntary sterilization has become a national policy and the centers of excellence are expanded to other high demand areas such as West Java and South Sumatera. The GOI will most certainly expand project developed IPCC module throughout the country. Serious discussions are being conducted between BKKBN and the Ministry of Health to expand the Problem Based Learning modules to midwifery schools outside project areas. In the adolescent reproductive health component, the peer educator and peer counselor approach is now adopted as the national strategy for ARH program. Government stakeholders are more aware about the role of NGOs and adolescents in ARH program, which was not true before the project. In the last couple of years, many ARH information and counseling centers were established throughout the country using the approach and models developed in the project. Summary and Recommendations l The project has demonstrated that reproductive health and maternal health issues are intersectoral and need to be implemented through a wide sector approach. Integrated planning particularly at the district level is a critical factor for the success of reproductive health and maternal health programs. l Project design and concept leading to project implementation through inter-sector partnership approach and a balance between demand creation activities and efforts to improve the quality of supply of maternal health services is appropriate for achieving optimal project objectives and results, not only in Safe Motherhood-Making Pregnancy Safer program, but also in other health programs. l Under the wide sector approach, the role of central government will focus on providing general policy and strategy guidelines, suggest appropriate interventions to be selected by the local government based on local situation and provide technical assistance, monitoring and evaluation. l District block grants combined with matching funds from the local government is a strategic way to improve program sustainability post project and to improve local government ownership to the program. l Collaboration among donors should be continued and strengthened in the future in order to increase efficiency and effectiveness of activities. (b) Cofinanciers: (c) Other partners (NGOs/private sector): 10. Additional Information - 21 - Annex 1. Key Performance Indicators/Log Frame Matrix Outcome/Impact Indicators:° Indicator Baseline 2002 2003 2004 Endline Improving utilization of quality MH services % increase in deliveries assisted by BDD, and by trained health providers* Kab. Cilacap 36.45 54.82 Kab. Rembang 41.65 51.55 Kab. Jepara 46.81 57.28 Kab. Pemalang 43.33 64.78 Kab. Brebes 40.63 48.60 Kab. Trenggalek 43.59 69.33 Kab. Jombang 72.98 90.80 Kab. Ngawi 70.45 84.85 Kab. Sampang 11.27 25.73 Kab. Pamekasan 25.04 33.50 At least 60% of cases with obstetric complications referred to health facilities providing emergency obstetric care % Mothers with complications referred to BEONC/skilled deliveries** No data 0.85% 4.96% 9.17% % Mothers with complications referred to CEONC/skilled deliveries** No data 2.17% 1.93% 2.86% Strengthening the sustainability of MH services at the village level No BdD TPC pilot Completion and evaluation of BdD pilot project, contracted under completed and especially TPC contract TPC scheme evaluated in 2002 Improving demand for and access to FP services % increase in a "quality index" of FP services No data - - No data Preparing adolescent to lead a healthy reproductive life % increase of knowledge about RH issues among adolescent Baseline*** Endline*** Male Female Male Female Discuss RH issues in general 60.1% 57.9% 72% 66% Discuss sexuality at home 17.0% 33.0% 30% 57% Ever heard HIV/AIDs 78.8% 81.7% 81% 97.5% Know about STDs 28.8% 14.3% 47% 34.3% Know about anemia 22% 32% 84% 95% Seeking counselling before marriage 9.2% 9.2% 56% 64% Etiology of HIV 15% 12% 53% 63% Women fertile period 33% 46% 58.5% 89% Women menstrual cycle 23% 56% 53.5% 63% Note: ° Project performance indicators linked to DOs. Data refer to 10 districts only. * This is the percentage of Birth Attendance by Doctor/Midwife for Children Age <5 year in 10 project districts. The baseline is Susenas 1999 and Endline is Susenas 2004. **The data is cumulative as of June of the FY from the project monitoring indicators (facility based information). ***Baseline study on RH service needs assessment by LD-UI, Endline is from rapid surveys. - 22 - Output Indicators:* Indicator Baseline 2002 2003 2004 Endline MH Services Increase technical skills of village midwives for providing MH services % BdDs trained in normal delivery 100%, except in Ngawi (60%) and care 0% Trenggalek (80%) % BdDs at remote villages trained 76.5% in life saving skills 0% Increase utilization of referral services for obstetric and neonatal emergency % of cases with ON complications who referred to emergency services No data 2.67% 3.22% 3.43% % Mothers with complications referred to BEONC/skilled deliveries No data 0.85% 4.96% 9.17% % Mothers with complications referred to CEONC/skilled deliveries No data 2.17% 1.93% 2.86% Increase quality and management of emergency care at HC and hospital % Caesarean Section of all deliveries in hospital No data 29.97% 25.79% 20.63% % deaths of all deliveries in hospital No data 1.09% 1.01% 0.68% Increase knowledge of the danger signs Knowledge of danger signs of 5 CJ CJ non 5 EJ EJ non maternal complications** districts project districts project Prolonged labour No data 6.7% 8% 10.2% 9% Haemorrage No data 22.5% 17.5% 16.3% 20% Fever No data 1.2% 1.3% 1.7% 2.5% Convulsion No data 2.3% 2% 1.2% 1.1% Malposition No data 8.7% 5.5% 12% 13.5% Knowledge of danger sign of 5 CJ CJ non 5 EJ EJ non deliveries among midwives** districts project districts project Premature Rupture of Membrane No data 13.3% 9.6% 15.9% 17.2% Ante and Post Partum Haemorrage No data 18.3% 16.3% 14.6% 17.2% Fever No data 1.15% 1.14% 1.12% 1.15% Prolonged Labour No data 6.7% 5.9% 3.9% 3.3% Convulsion No data 5.5% 3.5% 3.7% 5.1% Retained Placenta No data 7.6% 5.6% 5.6% 7.2% - 23 - Sustainability of MH services TPC is adopted TPC pilots evaluated by Implement pilot projects for 101 BdDs and financed by Lipiduta in 2002, by alternative financing of BdDs as participating in the local Unair in East Java in private providers the pilot governments 2002 Women of poor families delivering Central Java in 205 of 276women used babies attended by skilled providers 1999: 26.1% coupons*** Improved Quality of FP services Increased technical skills of FP providers Public & private providers trained in clinical FP standard 0 31.7% Provider trained in FP management 0 37.6% FP Workers trained in FP management 0 44.3% Improved counseling skills of FP providers, field workers and caders Midwives trained in IPCC 0 100% FP Workers trained in IPCC 0 100% Improved awareness of side effects and complications of FP services CJ: 0.11%, EJ: CJ: 0.02%, CJ: 0.01%, % Complications (side effects) 0.82% EJ:0.27% EJ: 0.2% CJ: 0.03%, EJ:0.18% CJ: 14.2%, EJ: CJ:8.8%, EJ: % unmet need for contraception No data 10.46% 15.3% CJ:13.9%, EJ: 9.9% Adolescent reproductive health Assessment Assessment of service needs for study conducted ARH by LDUI in 1999 - - Increased counseling services for adolescent CJ: 1299, EJ: CJ:2550, # of ARH facilitators 0 2405 EJ:8310 CJ: 2849 ,EJ:8900 CJ:541, # of peer educators 0 CJ: 262, EJ:575 EJ:1077 CJ: 658, EJ:1107 CJ:342, # of peer counselors 0 CJ: 117, EJ: 278 EJ:552 CJ: 437, EJ: 552 # ARH service & information CJ: 589, EJ: centers 0 CJ: 235, EJ: 185 373 CJ: 807, EJ: 444 Note: * Project performance indicators linked to Project components. Data refer to 10 pilot districts only (cumulative figures in June of the fiscal year). **10 district SMPFA evaluation as part of the 2003 IDHS. ***Lipiduta TPC evaluation in 2002. - 24 - Annex 2. Project Costs and Financing Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) Expenditure Category Procurement Method ICB LIB Other Total Cost* Investment Costs Equipment 1,390 927 748 3,064 Vehicles - - 68 68 IEC & Instructional Materials 2,808 6,335 2,340 11,484 Technical Assistance - - 3,705 3,705 Training and Workshops - - 13,694 13,694 Fellowships Local - - 733 733 Overseas - - 1,959 1,959 Evaluation and Research - - 5,897 5,897 DHS - - 1,400 1,400 Financial Support for BDDs - - 389 389 BDD Pilots - - 2,203 2,203 Civil Works/ furniture - - 336 336 Project Administration - - 1,167 1,167 Subtotal Investment Costs 4,197 7,262 34,639 46,099 Recurrent Costs Emergency Obstetric Services - - 4,703 4,703 Monitoring and Supervision - - 937 937 Subtotal Recurrent Costs - - 5,640 5,640 Total Base Costs 4,197 7,262 40,279 51,739 Physical Contingencies 210 363 2,014 2,587 Price Contingencies 617 1,067 5,920 7,604 Total 5,024 8,692 48,213 61,930 - 25 - Project Costs by Procurement Arrangements (LATEST Estimate) (US$ million equivalent) Expenditure Category Procurement Method ICB LIB Other Total Cost* Investment Costs Equipment - 594.04 1,040.37 1,634.40 Vehicles - - - - IEC & Instructional Materials 248.00 271.37 3,242.83 3,762.20 Technical Assistance - - 1,338.43 1,338.43 Training and Workshops - - 5,825.98 5,825.98 Fellowships Local - - 239.32 239.32 Overseas - - 989.82 989.82 Evaluation and Research - - 3,374.52 3,374.52 DHS - - 813.98 813.98 Financial Support for BDDs - - - - BDD Pilots - - 671.43 671.43 Civil Works/ furniture - - - - Project Administration - - 1,681.80 1,681.80 Contraceptives 5,066.36 - - 5,066.36 Safe Motherhood Grants - - 6,925.82 6,925.82 Subtotal Investment Costs 5,314.36 865.41 26,144.28 32,324.05 Recurrent Costs Emergency Obstetric Services - - 25.76 25.76 Monitoring and Supervision - - 116.86 116.86 Subtotal Recurrent Costs - - 142.62 142.62 Total Base Costs 5,314.36 865.41 26,286.90 32,466.66 Physical Contingencies Price Contingencies Total 15,943.07 2,596.22 78,860.70 32,466.66 - 26 - Project Costs by Component (US$ million equivalent) Appraisal Estimate* Actual Estimate % of Appraisal Component Total Total Total IBRD GOI IBRD GOI IBRD GOI (Bank+Gvt.) (Bank+Gvt.) (Bank+Gvt.) Component A. East Java Component A.I. Improving MH status, utilization and sustainability of MH 7,691.20 3,510.80 11,202.00 3,589.35 513.59 4,102.94 47% 15% 37% services I.a. Increasing Demand for MH Services 4,370.15 1,994.85 6,365.00 1,353.47 202.24 1,555.71 31% 10% 24% I.b. Improving Supply of MH services 2,679.77 1,223.23 3,903.00 1,767.43 311.35 2,078.77 66% 25% 53% I.c. Strengthening MH services at village level 641.28 292.72 934.00 468.45 - 468.45 73% 0% 50% A.II. Improving Quality of Family Planning Services 3,226.29 1,472.71 4,699.00 2,127.23 355.28 2,482.50 66% 24% 53% A.III. Preparing Adolescents to lead a healthy Reproductive 1,294.22 590.78 1,885.00 1,227.20 136.20 1,363.41 95% 23% 72% Life A.IV. Project Administration 377.63 172.37 550.00 1,607.37 106.83 1,714.20 426% 62% 312% Subtotal East Java 12,589.34 5,746.66 18,336.00 8,551.14 1,111.90 9,663.04 68% 19% 53% Component B. Central Java Component B.I. Improving MH status, utilization and sustainability of MH 7,837.44 3,577.56 11,415.00 2,880.85 384.90 3,265.76 37% 11% 29% services I.a. Increasing demand for MH services 4,484.13 2,046.87 6,531.00 1,029.78 160.67 1,190.45 23% 8% 18% I.b. Improving supply for MH services 2,712.04 1,237.96 3,950.00 1,488.00 224.24 1,712.23 55% 18% 43% I.c. Strangthening MH services at village level 641.28 292.72 934.00 363.08 - 363.08 57% 0% 39% B.II. Improving quality of family planning services 2,690.75 1,228.25 3,919.00 1,808.66 346.36 2,155.02 67% 28% 55% B.III. Preparing adolescents to lead a healthy reproductive life 1,292.85 590.15 1,883.00 1,329.08 145.32 1,474.40 103% 25% 78% B.IV. Project administration 377.63 172.37 550.00 1,410.18 24.60 1,434.78 373% 14% 261% Subtotal Central Java 12,198.67 5,568.33 17,767.00 7,428.77 901.19 8,329.96 61% 16% 47% Component A. A.I. Improving MH status, utilization and sustainability of MH 15,528.63 7,088.37 22,617.00 6,470.20 898.50 7,368.70 42% 13% 33% services I.a. Increasing Demand for MH Services 8,854.28 4,041.72 12,896.00 2,383.25 362.91 2,746.16 27% 9% 21% I.b. Improving Supply of MH services 5,391.80 2,461.20 7,853.00 3,255.42 535.59 3,791.01 60% 22% 48% I.c. Strengthening MH services at village level 1,282.55 585.45 1,868.00 831.53 - 831.53 65% 0% 45% A.II. Improving Quality of Family Planning Services 5,917.04 2,700.96 8,618.00 3,935.89 701.63 4,637.52 67% 26% 54% A.III. Preparing Adolescents to lead a healthy Reproductive 2,587.08 1,180.92 3,768.00 2,556.28 281.53 2,837.81 99% 24% 75% Life A.IV. Project Administration 755.25 344.75 1,100.00 3,017.54 131.43 3,148.98 400% 38% 286% Subtotal Component A 24,788.00 11,315.00 36,103.00 15,979.91 2,013.09 17,993.00 64% 18% 50% Component C. Central Level Component C.I. Technical Support/Training 5,658.20 2,582.80 8,241.00 10,938.97 924.68 11,863.65 193% 36% 144% C.II. Policy, reasearch and evaluation 3,987.04 1,819.96 5,807.00 4,048.85 341.74 4,390.59 102% 19% 76% C.III. Project Administration 1,090.31 497.69 1,588.00 1,498.94 28.19 1,527.13 137% 6% 96% Subtotal Central Level 10,735.54 4,900.46 15,636.00 16,486.76 1,294.61 17,781.37 154% 26% 114% Total BASELINE COSTS 35,523.55 16,215.45 51,739.00 32,466.67 3,307.70 35,774.37 91% 20% 69% Physical Contingencies 1,776.21 810.79 2,587.00 Price Contingencies 5,220.84 2,383.16 7,604.00 Total PROJECT COSTS 42,520.60 19,409.40 61,930.00 32,466.67 3,307.70 35,774.37 76% 17% 58% - 27 - Annex 3. Economic Costs and Benefits Not Applicable - 28 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 10/23/1995 3 POP.& HEALTH SPECIALI (1); S S SENIOR POP.& REPROD.HEALTH ADVISOR (1) PRINCIPAL ECONOMIST (1) 21/06/1996 4 POP.& HEALTH SPECIALI (1); S S REPROD.HEALTH ADVISOR (1) MATERNAL HEALTH AND EVALUATION SPECIALIST (1) OPERATIONAL OFFICER (1) Appraisal/Negotiation 20/01/1997 4 POP.& HEALTH SPECIALI S S (1); SENIOR HEALTH SPECIALIST (1) HEALTH ECONOMIST (1) OPERATIONAL OFFICER (1) Supervision 11/17/1997 4 POP.& HEALTH SPECIALI S S (1); SENIOR HEALTH SPECIALIST (1) HEALTH ECONOMIST (1) OPERATIONAL OFFICER (1) 12/05/1997 2 POP. & HEALTH SPECIALI S S (1); OPERATIONS OFFICER (1) 04/30/1998 4 HEALTH/POP SPECIALIST (1); S S OPERATIONS OFFICER (2); FINANCIAL MGT CONSULT (1) 08/14/1998 5 HEALTH/POP SPECIALIST (1); S S OPERATIONS OFFICER (2); MAT. HEALTH CONSULTANT (1); IEC CONSULTANT (1) 02/14/1999 4 HEALTH/POP SPECIALIST (1); S S OPERATIONS OFFICER (2); CONSULTANT (1) 10/16/1999 5 HEALTH & POP SPEC. (1); S S OPERATION OFFICER (2); SAFE MOTHERHOOD CONS. (1); FINANCIAL MANAG CONS. (1) - 29 - 07/29/2000 3 HEALTH/POPULATION SPEC S S (1); OPERATIONS OFF/HEALTH (1); OPERATION OFF/IMPL. (1) 10/20/2000 3 PUBLIC HEALTH, TTL (1); S S PUBLIC HEALTH,OPERATNS (1); MATERNAL HEALTH (1) 09/10/2001 4 PUBLIC HEALTH, TTL (1); S S HEALTH, DEPUTY TTL (1); FINANCIAL MANAGEMENT (1); PROCUREMENT (1) 05/03/2002 6 PUBLIC HEALTH, TTL (1); S S HEALTH, DEPUTY TTL (1); TEAM ASSISTANT (1); MONITORING AND HMIS (1); MATERNAL HEALTH SPEC (1); REPROD H. SERV.QUALITY (1) 09/25/2002 6 TASK TEAM LEADER (1); S S HEALTH SPECIALIST (1); QUALITY SPECIALIST (1); OPERATIONS OFFICER (1); FINANCIAL MANAGEMENT S (1); PROCUREMENT SPECIALIST (1) 09/25/2002 6 TTL, HEALTH SPECIALIST S S (1); HEALTH OPERATIONS (1); PROCUREMENT SPECIALIST (1); FINANCIAL MAN. SPEC (1); QUALITY AND FP PROGRAM (1); WHO MATERNAL CARE SPEC (1) 02/18/2004 7 PUBLIC HEALTH (1); HEALTH S S OPERATIONS (1); FINANCIAL MANAGEMENT (1); PROCUREMENT (1); SERVICE QUALITY (1); MATERNAL HEALTH DEMAND (1); HEALTH SPECIALIST (1) 09/30/2004 3 TASK TEAM LEADER (1); S S CONSULTANT (2) (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation N/A 247 Appraisal/Negotiation N/A N/A Supervision N/A 530 Total N/A 778 - 30 - - 31 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA Project Components Improving the Maternal Health Status and Health Services at the Village Level HS S U HU Increasing the Demand for and Access to High Quality Family Planning Services HS S U HU Preparing Adolescents for Healthy Reproductive Lives by Increasing Their Knowledge of Reproductive Health Issues HS S U HU Safe Motherhood Grants HS S U HU Technical Support and Training for Central and Provincial Level Staff HS S U HU Policy, Research and Evaluation Activities including: (a) Policy Studies; (b) Monitoring and Evaluation Activities; and TA to Various Implementing Agencies for Monitoring and Evaluation Activities HS S U HU Project Administration Support including Project Management Units at Central and Provincial Levels HS S U HU - 32 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU - 33 - Annex 7. List of Supporting Documents Health Nutrition and Population Unit East Asia and Pacific Region. Indonesia Safe Motherhood Project: A Partnership and Family Approach, Project Reference Documents World Bank (1997a). "Project Appraisal Document on a Proposed Loan in an Amout of US$42.5 Million to the Republic of Indonesia for A Safe Motherhood project: A Partnership and Family Approach, " Report No. 16624-IND (June 3, 1997) World Bank (1997b). "Loan Agreement (Safe Motherhood Project: A Partnership and Family Approach) between Republic of Indonesia and International Bank for Reconstruction and Development," Loan Number 4207 IND(August 4, 1997) World Bank (1997-2004). Complete Aide Memoires World Bank (1997-2004). Complete Project Status Reports World Bank (1999-2005). Amendments to the Loan Agreement BKKBN, DEPKES, World Bank. "Safe Motherhood: Partnership & Family Approach (Experiences from SMPFA Project)," Jakarta, 2004 Knowles, James C., "Consultant's Report of Technical Assistance Provided to the BDD Sustainabliity Component of the Safe Motherhood Project (15-26 May, 2000), Draft: 14.6.00 Kushadiwijaya, Haripurnomo (Consultant), "Advocacy of Exit Strategy of SMPFA Priject, Final Report", not dated PT.Gama Multi Usaha Mandiri, "Final Evaluation Safe Motherhood Project: A Partnership and Family Approach (SMPFA)" Final Report, not dated - 34 - - 35 -