Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) Report Number : ICRR0021282 1. Project Data Project ID Project Name P111432 RCIP - Phase 3 Country Practice Area(Lead) Africa Transport & Digital Development L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46010,IDA-46020,IDA-46140 28-Feb-2015 144,692,369.64 Bank Approval Date Closing Date (Actual) 25-Jun-2009 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 151,000,000.00 0.00 Revised Commitment 151,000,000.00 0.00 Actual 144,833,459.25 0.00 Prepared by Reviewed by ICR Review Coordinator Group Nestor Ntungwanayo Peter Nigel Freeman Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives • The program is the third phase of a multi-country Regional Communication Infrastructure Program (RCIP) approved in March 2007, which was implemented through three projects in Tanzania, Mozambique, and Malawi. The program had a set of identical objectives for Tanzania and Mozambique, and specific objectives for Malawi as presented below: • PDOs for Tanzania and Mozambique: As per the Financing Agreement for each country on p. 5, "the objectives of the Project are to support the Recipient’s efforts to: (i) lower prices for international capacity Page 1 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) and extend the geographic reach of broadband networks; and (ii) improve the government’s efficiency and transparency through eGovernment applications." • PDO for Malawi: As per the Financing Agreement on p.5, "the objective of the Project is to support the Recipient’s efforts to improve the quality, availability and affordability of broadband within its territory for both public and private users." • The statement of PDOs in the respective Financing Agreements was identical to the objectives as stated in the respective Project Appraisal Document -called Technical Annexes - prepared under this program. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Project's components as described in the respective FAs are summarized below: Component 1: Enabling Environment, including Monitoring and Evaluation Capacity-building (US$19.43 million at appraisal, (US$19.43 million, with actual cost of US$12.3 million) ) • Tanzania (US$14 million, with actual cost of US$6.0 million) This component aimed to: (i) build the capacity of the Ministry of Communication Science and Technology (MCST) and other key agencies involved in the ICT Sector for monitoring and evaluation by providing the related TA and capacity building activities; (ii) strengthen the policy and regulatory environment; and (iii) promote sector reform in order to maximize the benefits of access to international bandwidth. Key activities included the following: (a) strengthening the capacity for ICT policy making; (b) building the capacity for an ICT agency; (c) implementing a universal access strategy; (d) designing contracts for the pre-purchase of international capacity; (e) preparing bidding documents for the Government Communications Network; (f) building the capacity for certification of trainers to enhance use of ICT within the Recipient’s government; (g) building the capacity for managing the changing environment and fast technology evolution; (h) building the capacity for the development and implementing the national ICT infrastructure development program; and (i) airing information, education and communication messages. • Mozambique (US$3.43 million, with actual cost of US$4.3 million) Page 2 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) This component was to support the following activities: (i) building the capacity of key agencies involved in the ICT sector toward improving its overall competitive environment including provision of technical advisory services and training; (ii) providing capacity support for: (a) implementation of recommendations for the Environmental and Social Management Framework (ESMF) and Resettlement Policy Framework (RPF); (b) implementation of monitoring and evaluation framework and systems; (c) development and implementation of a communications strategy and establishing channels of communication; and (iii) the provision of technical advisory services for capacity building activities. • Malawi (US$2.0 million at appraisal, with actual cost of US$2.0 million) This first sub-component aimed to build the capacity of the Privatization Commission (PC), the Ministry of Information and Civic information Education (MICE), the Malawi Communications Regulatory Authority (MACRA) and other key agencies involved in the ICT Sector for monitoring and evaluation, and implementation toward improving the overall competitive environment for the ICT sector through the following specific activities: (a) to support the negotiations with private and public agencies in the development and operation of cross-border connectivity; (b) to maximize the benefits under access to capacity and to assess the policy and regulatory environment; (c) to support the policy and regulatory capacity, consensus building activities, and strengthening the policy making role of the MICE; (d) to draft a new ICT legislation and develop a corresponding regulatory framework; (e) to develop a legal framework supporting the development of public private partnerships arrangements in ICT sector; and (f) to support monitoring and evaluation of the Project activities. The second sub-component was to carry out an analysis of potential environmental and social impacts of the Project, including monitoring and evaluation of the mitigation measures that may be implemented pursuant to the Environmental and Social Management Framework (ESMF) and the Resettlement Policy Framework (RPF), through the provision of technical advisory services. The third sub-component aimed to carry out the overall communications strategy including supporting: (i) airing communication messages to promote proactive engagement of the civil society and other ICT- interest groups; (ii) timely disclosure, dissemination and implementation of the recommendations specified in key documents; (iii) institutions involved in upgrading of ICT infrastructure; and (iv) the private sector engagement in ICT. Component 2: Connectivity (US$89.1 million, with actual cost of US$91.9 million) • Tanzania (US$60 million at appraisal, with actual cost of US$63.0 million) Page 3 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) Under this component, identified activities under the national ICT infrastructure development program were as follows: (i) supporting pre-purchase of international bandwidth for priority targeted user groups, (ii) supporting the virtual government communications network, and (iii) supporting the extension of coverage and access to ICT services in rural areas using public private partnership arrangements. • Mozambique (US$14.6 million, with actual cost of US$15.5 million) This component has three sub-components as follows: (i) to establish a national internet exchange point, or strengthen the existing one with the necessary technology infrastructure and institutional capacity to provide services to the internet service providers, including carrying out a study to develop a viable business model; (ii) to purchase Telecom Services on the National Backbone and on international networks for targeted users; and (iii) to support the extension of GovNet to institutions at central and provincial levels, and introduce district level institutions to the network, through the acquisition of equipment • Malawi (US$14.5 million, with actual cost of US$13.4 million) The connectivity component had three categories of activities aimed to: (i) purchase international connectivity capacity to support the development of regional communications links with coastal landing stations of submarine cables, including the provision of technical advisory services, studies, legal advice and transaction support for a public private partnership, (ii) support connectivity for selected public institutions and public internet access points, and (iii) support the establishment and operation of a neutral point of interconnection to allow equal access to regional connectivity by the private sector and other operators. Component 3: Transparency- eGovernment applications (US$32.4 million at appraisal, with actual cost of US$26.6 million) • Tanzania (US$22.0 million, with actual cost of US$20.0 million) Supporting the implementation of targeted eGovernment applications had activities aimed to: (i) develop a national business portal as a one-stop shop for all business registration information; (ii) develop a telemedicine system for Muhimbili National Hospital; (iii) scale up the national vital registration system; (iv) enhance accessibility of land records through websites and mobile phone text messages, (v) develop an eProcurement pilot for the Medical Stores Department, (vi) support a selected additional eGovernment application; and (vii) support the capacity of the PO-PSM and other key agencies involved in rolling-out of eGovernment, and airing of related information, education and communication messages. Page 4 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • Mozambique (US$10.4 million, with actual cost of US$6.6 million) This component had three sub-components as detailed below: (i) to support the Recipient government's cross-cutting activities for operation of common hardware, software and communications, including: (a) defining and establishing appropriate technical standards, data models and procedural schemes to enable seamless interoperability of the eGovernment applications platforms; (b) establishing a web-based portal platform and user interface of the eGovernment services; (c) establishing common desktop standards for hardware, software and applications/licenses and local support services; (d) establishing a government Enterprise Service Bus to support messaging and transfer of data between applications and lower levels of computing; (e) supporting the design and the operating plan for an eGovernment services operations center; and (f) building the capacity for eGovernment services and platforms; (ii) to support targeted eGovernment applications including: (a) carrying out a concept and strategy for the establishment of a civil identity registry system, as well as a needs assessment for the overall design; (b) preparing bidding documents to facilitate private sector participation; and (c) developing a civil identity registry; and (iii) to support the land information management system, including: (a) building the human resource and institutional capacity for use of the land information management systems and graphical information system; (b) strengthening the capacity for technical Project management skills for the rollout of the system; and (c) integrating the system into the eGovernment platform and associated equipment. Component 4: Project Management Support (US$7.84 million at appraisal, with actual cost of US$11.7 million) • Tanzania (US$4 million, with actual cost of US$4.0 million) This component intended to support the Ministry of Communication, Science and Technology (MCST) and President's Office-Public Service Management (PO-PSM) capacity for management and coordination of project activities, including procurement, financial management, monitoring and evaluation, communication, and environmental and social safeguards. • Mozambique (US$2.34 million,with actual cost of US$3.2 million ) Page 5 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) This component aimed to strengthen the capacity of the Ministry of Science and Technology (MCT), the Ministry of Transport and Communication (MTC) and the Project implementing Unit ( PIU) for management and coordination of project activities, including the recruitment of the Project Coordinator, an advisor at the MTC, a lead information technology architect, a procurement manager, a financial management specialist, a monitoring and evaluation specialist, a communications specialist, and environmental consultants through the provision of technical advisory services, as well as acquisition of goods and equipment. • Malawi (US$3.5 million, with actual cost of US$4.5 million) This component was to support the project's coordination capacity by the recruitment of a project manager, a technical expert, a procurement specialist, and a financial management specialist, through the provision of technical advisory services, and the acquisition of goods. A budget amount for price contingency (US$1.5 million) was part of this component. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost: The total cost of the project amounted to US$143.0 million or 94 percent of the approved amount, which funded activities in Tanzania, Mozambique and Malawi in the respective amounts of US$93.0 million, US$29.9 million, and US$19.9million. Financing: The project was funded by a total credit in the amount of US$151 million equivalent provided to Tanzania, Mozambique and Zambia, in the respective amounts of US$100.0 million, US$31.0 million, and US$20.0 million equivalent. Borrower Contribution: There was no expected borrower contribution, and none occurred during project implementation. Dates: The program was approved on June 29, 2009 and made effective on October 14, 2009, with an original closing date of February 28, 2015. The sub-projects were restructured as follows: • the sub-project for Tanzania was restructured twice, the first time in January 2015 to extend the closing date to December 31, 2016, and the second time in November 2016 to slightly revise the results framework, and extend the closing date to December 2017. • the sub-project for Mozambique was restructured twice, the first in August 2010 to provide a waiver easing the conditions of withdrawal of proceeds for a given category of contracts not exceeding 10% of the value allocated to that category, and the second in February 2015 to revise a number of intermediate indicators, and to extend the closing date to June 30, 2016. Page 6 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • the sub-project for Malawi was restructured in November 2013 to integrate the following changes: (i) an extension of the closing date by 16 months to June 30, 2016, (ii) a reallocation of credit proceeds between budget categories, and (iii) adjustments in the disbursement projections to account for a longer implementation period. • The program was closed on June 30, 2016, 16 months after the original closing date. 3. Relevance of Objectives Rationale At closing, the RCIP3 is still highly relevant to the World Bank's regional strategy and country-specific strategies. The program is consistent with Priority Area 4 in the 2018 Africa Strategy, which identifies regional integration as one approach to improve connectivity, leverage economies of scale, and enhance productivity. The program also supports the Regional Integration and Cooperation Assistance Strategy (FY18-FY23) targeting sustained investment in regional ICT connectivity in the Africa Region. The sub-project in Tanzania was highly relevant to the development objectives identified in the FY18-22 Tanzania Country Partnership Framework (CPF). The connectivity component was instrumental to achieve the objective 1.6 aimed at enhancing transport, energy and digital connectivity for improved services to rural areas, under Focus Area 1. The eGovernment component is key to achieving the objective 3.3 aimed at better leveraging ICT to modernize the public sector under Focus Area 3. The sub-project in Mozambique was still highly relevant to the development objectives of the Country Partnership Strategy FY12-15, which was extended to FY16. Pillar I devoted to competitiveness and employment considers improved access to affordable telecommunications services, particularly Internet services both fixed and mobile, as an essential ingredient to improve access to information, doing business, and effective decentralization of services to the provinces. A more recent CPF for the period FY17-21 maintains that the connectivity component remains relevant to the CPF’s objective of “Promoting Diversified Growth and Enhanced Productivity”, and its Transparency Component remains relevant to the CPF’s objective of “Increasing Accountability and Transparency of Government Institutions”. In Malawi, the most recent World Bank's Country Assistance Strategy (FY13-FY16, extended to FY17) identifies ICT as a priority sector to promote sustainable, diversified, and inclusive growth (Theme 1 of the CAS), specifically through the diffusion of reliable broadband connections throughout the country. Overall, the RCIP3 is highly relevant to the WBG's integration strategy in Africa, and to the development priorities of the beneficiary countries. Page 7 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective (i) To lower prices for international capacity and extend the geographic reach of broadband networks. Rationale Theory of change • Causality: The causality chain underlying the first objective of the RCIP3 was not discussed in the original PAD and respective Technical Annexes for Tanzania and Mozambique. There was only a result framework and monitoring at country level. While the ICR doesn’t discuss the theory of change as such, it provides in Figure 1 and in the Annex 1.B. a result chain showing the linkages between the project interventions, the key assumptions, the outputs generated, and the expected outcomes and long-term development goals. Overall, the results chain is sound and supports the logical relationship between the project interventions and the achieved outcomes in each country. For instance, efficient connectivity requires Government support in the acquisition of key communication infrastructure, Government policies favoring private sector buy-in and investment, and adequate technical capacity. • Attribution and Counterfactual: While the ICR does not discuss specifically the attribution issue, it underscores the role played by respective Governments, the private sector and other external partners in complementing the role played by this program in the achieved outcomes. In Tanzania, the Government played a key role in creating a conducive environment (through the backbone infrastructure in particular) allowing the private entrepreneurs to invest in the ICT sector, and prior support to Tanzania by the World Bank and China was critical. In Mozambique, support to the ICT sector was a coordinated effort that included the Government and bilateral and multilateral cooperation. Overall, the financial contribution of the RCIP3, and the convening role of the World Bank played a key role in achieving project outcomes, but other domestic and external partners played a role too. The ICR did not discuss the counterfactual issue. Below is a summary of the program's efficacy, reflecting elements detailed in the Section II. B and the results framework in Annex I of the ICR. Tanzania: Outputs Page 8 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • A National ICT Broadband Backbone (NICTBB) network was built in accordance with technical specifications; • Completion of Public Private Partnership (PPP) infrastructure investments in rural areas directly responsible for the extension of connectivity to 347 rural wards and 1,393 villages. Outcomes All targets for the 4 PDO indicators were achieved or surpassed, and targets for 5 out of 7 intermediate outcome indicators were met (see Annex 1 of the ICR): • 51,512 Mbps of international Internet bandwidth were acquired up from 302 Mbps at baseline, exceeding the revised target of 50,000 Mbps; • The international wholesale price for an E1 capacity link dropped from US$10,000 per month at baseline (2008) to US$70 per month at project closing (2017), against a revised target of US$100.00 and the average monthly retail price of 1 Mbps Internet access went down from US$800 (2008) to US$16 (2017). The drop in prices went hand in hand with the increase in coverage. The wholesale monthly price per Mbps went from US$52.96 (first government purchase in 2012) to US$33.33 in 2017. The availability of more affordable wholesale Internet bandwidth was leveraged by private telecom operators in Tanzania to increase the affordability of retail prices. • Coverage of mobile networks increased from 65 percent (2008) to 94.5 percent (2017). Total teledensity (fixed and mobile) increased from 32 percent (2008) to 80 percent (2017), achieving the revised target. • Internet user penetration jumped to 40 percent, achieving the revised target against a level of 0.98 percent at baseline. An additional 2.6 million Tanzanians from remote rural areas gained access to connectivity; Mozambique: Outputs • Introduction of new regulation and improved competition in the ICT sector, which laid the foundations for increased access to online public services; • Completion of: (i) telecommunication and broadband strategies, (ii) updated legislation for the telecommunication sector, (iii) regulatory tools for licensing, and (iv) cost modeling of telecommunication services, (v) migration from analog to digital TV in the cities of Maputo, Beira, Quelimane, Nampula, Pemba and Tete, (vi) Strategic Telecommunications Infrastructure Map & Database (SGIT), and (vii) award of the third mobile operator license — Movitel; • 54 Community Media Centers (CMCs) in rural areas were connected with high quality internet service; Page 9 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • Provision of IT equipment, technical assistance and capacity building for the Land Information Management System in six provinces (Maputo, Gaza, Tete, Inhambane, Sofala, and Manica); • Connectivity and Network Equipment Contracts with 82 institutions linked through MoRENet and 16 wireless campuses; • Strategy for revenue generation and value added services; • Connection to UbuntuNET of 54 Community Media Centers equipped and connected to high quality internet service and a sustainable business model; • 21,000 citizens have received various forms of computer and other trainings at the CMCs (some to certification level); • 74,000 citizens with internet connectivity to access telemedicine services and diagnoses, agricultural extension and market information; • 123,945 citizens have accessed other services (photocopying, etc.); • MOZIX association was formally approved as an independent legal entity by the Ministry of Justice. Outcomes Out of the 12 intermediate indicators and targets, 2 were dropped, and 10 were achieved or surpassed (see Annex 1 of the ICR) and three out of the four PDO targets were exceeded, and the fourth was partially achieved as detailed below: • The international wholesale price for an E1 capacity link dropped from US$9,000 (2008) to US$782 per month (2016) or about 80 percent of the revised target. The average monthly retail price of 1 Mbps broadband Internet access went down from US$140 (2008) to US$30 (2016), and the average price of a monthly 3 Mbit/3G mobile data subscription dropped from US$33 (2008) to US$11 (2016); Countrywide, the number of telecommunications service providers increased from 7 (3 fixed and mobile, and 4 ISPs) in 2008 to 33 (5 fixed and mobile providers, and 28 ISPs) in 2016; • The internet use increased from a baseline of 1 percent in 2008, and reached 18 percent at completion, more than doubling the revised target of 8 percent; • Total teledensity increased from 20 percent (2008) to 69.7 percent at completion, exceeding the revised target of 60 percent; • The volume of international traffic also increased from 212 Mbit/s (2008) to 7,755 Mbit/s (2016), exceeding the target of 6500 Mbit/s. Access to mobile broadband reached 25.7 percent at project closing from 0.05 percent prior to the project. • Moreover, improved legislation, regulatory reforms, tools for licensing and cost modeling of telecommunications services led to increased competitiveness in the sector. Technical assistance related to the award of a 3rd license had a major impact on competition and investment in the mobile market, bringing along increased access to ICT, lower ICT costs, and better-quality services. By its first anniversary, the third operator (Movitel) had rolled out very extensive fixed and mobile network facilities with a strong rural focus that covered all 128 administrative districts of Mozambique, and free-of-charge connection of thousands of schools to the Internet; Page 10 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • Finally, increased competition and intensified rollout of cellular services in new areas of the country, as well as the upgrading of their network capacity, all led to improved quality of services, and lower retail costs. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve the government’s efficiency and transparency through eGovernment applications. Rationale Theory of change Identical as under objective 1. Below is the program efficacy, reflecting the achieveemnst detailed in the Section II. B and the results framework in Annex I of the ICR. Tanzania: Outputs • Completion of infrastructure to launch eGovernment, including the deployment of GovNet connecting 77 Local Government Agencies (LGAs) and 72 other Ministries, Departments and Agencies (MDAs); • Acquisition of bandwidth to provide subsidized Internet services to Government agencies, schools and hospitals, as well as key equipment for the Government Data Center; • Development of the Government Mobile Platform, a one-stop shop for online public services, and several high-development-impact eGovernment applications; • Strengthening of the Birth and Death Registration System (BRS) through a customized application for civil registration, and the supply of front-end systems for automated BRS in 60 registration sites in remote areas, plus 4 registration sites at the Registration, Insolvency and Trusteeship Agency (RITA); • The Telemedicine System connected three hospitals in each region to the Muhimbili National Hospital (MNH), a national referral hospital, and 129 patients served by the telemedecine system, up from 0 at baseline; • Enactment of the Cybercrimes Act I; • 149 Government institutions connected to Govnet Page 11 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • 2,850 Government staff trained; • 40 sites connected to the Vital Registrations System, up from 5 at baseline; • Installation of eOffice management system in 28 government institutions; • 60,000 archived documents accessible through eRecords; • A total of 20 radio and 30 TV programs were produced, aired, and uploaded online, contributing to increased visibility of Tanzania’s ICT policy and programs, and 41 programs aired to increase awareness about eGovernment applications; • 16,009,714 electronic records/events processed yearly with eGov apps, up from 0 at baseline. Outcomes All three PDO objectives were exceeded, including other sector improvements as detailed below: • 73.3% users highly satisfied or satisfied with the quality of digital public services with no data at baseline, largely exceeding the revised target of 40 percent; • The project supported the design and implementation of an eRecords system to ensure the management and preservation of digital records for government and business needs and citizen access. The volume of electronic records/events processed with eGovernment applications reached 16,009,714, reaching more than five times the revised target of 3,000,000, from a baseline of 0. • The direct project beneficiaries amounted to 2,619,110, nearly doubling the revised target of 1,320,000, with a baseline of 0. Other improved indicators include: (i) 3 minutes to fill in an official letter (down from 1 hour in absence of eGov), (ii) 2 minutes to search a document (down from 1 hour absence of eGov), and (iii) 1 hour to get Birth or Death registration (down from 7 days in absence of eGov) ; • The Unified National eProcurement System, supported through the development of a customized application software, and related training of administrators and users, decreased room for corruption in public procurement, thanks to services such as eTendering, ePurchasing, eContract, eDispute, eAuction, and ePayment; • The above system is integrated with other government systems (Integrated Financial Management Information System (IFMIS) at the Ministry of Finance and Planning, Short Message Service (SMS) gateway, bank payment systems, and Enterprise Resource Planning (ERP) at the Government Procurement Service Agency) to ensure cross-check of public procurement information; • The National Business Portal, now integrated with the Tax Identification Number verification system, and the government Electronic Payment Gateway, provides an online one-stop-shop to set up and run businesses in Tanzania. The eOffice system, currently tested in 28 government institutions, automated government daily business operations; Page 12 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • The project supported the design and implementation of an eRecords system to ensure the management and preservation of digital records for government and business needs and citizen access. More than 60,000 documents are now accessible online through this system. • The eGovernment Publicity Program increased awareness of the government’s ICT investments and progress on ICT policy in Tanzania and citizens’ awareness of digital applications and services available, as captured by the number of followers on the eGovernment social media pages (23,735 followers within a few months). Mozambique: Outputs • 425 government institutions connected to GovNet; • 6000 Microsoft licenses to the Government Institutions, and provision of IT equipment, technical assistance and capacity building for the Land Information Management System in six provinces (Maputo, Gaza, Tete, Inhambane, Sofala, and Manica); • Provision of eGovernment applications to Social Security (SISSMO), legal registration, infrastructure management system (SGIT), Tax filing (NUIT), Land registrations System (SiGIT), and Civil Registry (e- RCEV); • Deployment of GovNet connected 425 government agencies and supported the ongoing government decentralization process; • The Mozambique Research and Education Network (MoRENet) provided low-cost high-speed connection to 82 universities and research centers, in each of the country’s 10 provinces; • 2,701 births registered using e-RCEV25, and 733 from the capital city of Maputo; • Development of both SiGIT Mobile and SiGIT Cloud modules; • 54 Community Multimedia Centers (CMCs) in remote areas of the country had been connected to the Internet with bandwidth varying between 1.5 and 2 Mbps for download and 512 Kbps for upload, allowing for improved access to information and education; • Acquisition of equipment to support the interoperability of systems and databases across government, and capacity building. • Expansion of the national data center and the creation of e-BAU (electronic Balcão de Atentimento Único), an electronic one-stop shop platform for public services; • a feasibility study to promote interconnectivity among data centers and server rooms; and the purchase of software for government agencies was completed. Outcomes • Performance toward the user perception of quality of public services was not documented; Page 13 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • The volume of yearly electronic records/events processed through selected eGovernment applications increased from zero (2008) to 230,000 at completion, multiplying the revised target by close to ten times; • The direct project beneficiaries (out of which 50 percent are women) reached 1,065,957, almost ten times the revised target; • Other outcomes include the development of multiple eGovernment applications, accessible through: (a) the Civil Registry and Vital Statistics- e-RCEV25, which allows citizens to use mobile phones to register births and deaths, request certificates, and request personal identification cards; (b) the e-RCEV allows the Government to receive geo-mapping of such vital records; (c) the Land Information Management System (LIMS), which supports the Government program "Terra Segura" (Secure Land) to register 4,000 communities and 5 million land parcels and secure tenure rights and finally; and (d) the SiGIT application upgraded the information received for Community Delimitations and Individual Registrations and Regularizations, and established quality control procedures and standards to ensure data integrity and accuracy during processing. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective To improve the quality, availability and affordability of broadband within its territory for both public and private users. Rationale Theory of change • Causality Chain: The causality chain underlying the third objective of the RCIP3 was not discussed in the original PAD and the specific Technical Annex for Malawi. There was an annex laying out the result framework and monitoring at country level. While the ICR doesn’t discuss the theory of change as such, it provided in Figure 1 and in the Annex 1.B. a result chain showing the linkages between the project interventions, the key assumptions, the outputs generated, and the expected outcomes and long-term development goals. Overall, the results chain is sound and supports the logical relationship between the project interventions and the expected outcomes. • Attribution and Counterfactual: While the ICR does not discuss specifically the attribution issue, it underscores the role played by the Malawi Government, especially the role played by the private investement in the ICT sector and the technology innovations. Overall, the financial contribution of the Page 14 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) RCIP3, and the convening role of the World Bank played a key role in achieving project outcomes, but other domestic and external partners played a role. Both the ICR and this review don’t discuss the counterfactual issue, because of the dearth of adequate information and assumptions. Below is the program efficacy, reflecting elements detailed in the Section II. B and the results framework in Annex I of the ICR. Malawi Outputs • Creation of PPPs for regional connectivity; • Award of license for regional fiber links and a Virtual Landing Point to SimbaNET; • Western regional fiber link completed in August 2015; • Northern/Eastern fiber link and permanent Virtual Landing Point completed in Dec 2015; • 11,680 Mbps of international bandwidth, up from 180 at baseline; • 145 educational and government institutions in various districts supplied with last mile broadband connectivity; • The project provided technical and financial assistance to the deployment of a Virtual Landing Point (VLP) and the regional connectivity system to bring high-speed Internet from Tanzania and Zambia to Lilongwe and decentralized government agencies in the country; • Pre-feasibility studies were completed, and a PPP for regional connectivity and the pre-purchase of international bandwidth was created; • The Tanzanian firm SimbaNET completed the western regional fiber link through Zambia in August 2015 and the northern/eastern fiber link through Tanzania and permanent VLP in December 2015. • The installation of a permanent VLP and the completion of the regional connectivity network brought lowcost, high-end bandwidth international connectivity to Malawi for the first time. The open-access principle, a key provision of the entire RCIP program, was integrated into the Government’s contract with SimbaNET as well as into SimbaNET’s operating telecom license. Outcomes . All targets for the 4 PDO indicators, and targets for 4 out of 5 of the intermediate indicators were surpassed (see Annex 1 of the ICR). There was improved quality, availability, and affordability of broadband within Malawi for both public and private users, as illustrated by the following performance indicators. • The quality of broadband improved substantially as measured by the increase in international Internet bandwidth from 180 Mbps at baseline (2008) to 11,680 Mbps at project closing (2016), almost 30 times the target; Page 15 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • The broadband availability also improved significantly as internet user penetration increased from 0.7 percent (2008) to 15.7 percent (2016), almost 8 times the PDO target; • the price of a wholesale international capacity E1 link dropped from US$21,230 (2008) to US$962 (2016), largely exceeding the target of US$10,000 • Similarly, the price of wholesale international connectivity fell from about US$3000 per Mbit/s per month in 2013 to a notional cost of just US$135 for public users and US$85 for private users at project closing, and finally (iv) More market players accessed capacity on submarine cables, increasing from 0 at baseline (2008) to 8 at project closing. In addition, the number of operational Internet Service Providers (ISPs) increasing from 12 at baseline (2008) to 21 by project closing,slightly exceeding the target; • The above increases in retail market activity and maturity drove more competition, shown by a reduction in the mobile market of the Herfindahl-Hirschman Index (HHI), a standard measure of market concentration, from 5777 at project launch (Q2 2009) to 5029 at project closing (Q2 2016); • The increased competition, along with the lower wholesale bandwidth costs, resulted in price reduction. (iv) The average price of a mobile call dropped from US$0.20 at baseline (2008) to an average of US$0.11 by project closing. Reduced international connectivity costs had a marked impact in the Internet retail market as well. The retail price of Internet access dropped from US$120/MB at baseline (2008) to US$5.80/MB per month by project closing; • SimbaNET offered wholesale bandwidth to private operators via the VLP at the same discounted price that had been agreed for Government users, reducing a major portion of operators’ costs, enabling many more ISPs to compete in the market, thus having a downward effect on retail market prices. The project contributed to lowering the costs and increasing the availability of wholesale international bandwidth and transformed the ICT sector in Malawi. Rating High PHREVDELTBL PHOVRLEFFRATTBL Rationale (i) Program assessment: The program (RCIP3) had three objectives, two of which were implemented in Tanzania and Mozambique and were stated as follows: (i) " to lower prices for international capacity and extend the geographic reach of broadband networks, and (ii) "to improve the government’s efficiency and transparency through eGovernment applications". The third objective was implemented in Malawi and was stated as follows: "to improve the quality, availability and affordability of broadband within its territory for both public and private users". Performance toward the first objective as developed under Section 4 was Substantial because (i) the quasi- totality of PDO indicators was achieved or surpassed, and (ii) increased competition and intensified rollout of cellular services, and enhanced network capacity in Tanzania and Mozambique led to improved quality of services, and lower retail costs. Results toward the second objective as detailed under Section 4 was also Page 16 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) Substantial, because (i) the quasi-totality of PDO indicators were achieved or exceeded, and (ii) in Tanzania and Mozambique, eGovernment infrastructure was significantly scaled up, and eGovernment applications and usage exploded, leading to improved Government's efficiency and transparency in both countries. Achievements toward the third objective as described under Section 4 was high, because: (i) all PDO indicators were exceeded, and (ii) there was improved quality, availability, and affordability of broadband within Malawi for both public and private users at the project closure. On balance, based on the results achieved under each objective, the rating of the RCIP's efficacy is Substantial Attribution: The outcome achieved under this program is attributable to multiple factors which vary across countries as detailed in the table below. In Malawi, the RCIP was central to achieve the outcome, but in parallel, there were technology innovations in the ICT sector, and substantial private investment. In Mozambique, the RCIP created the conditions for private investments and competition in the ICT sector, but technology improvements and existing investments played a role. Finally, in Tanzania more factors played out to archived the outcome, including technology innovation, private investment, a loan from China, and a prior World bank project (PRAP) Table 1: Attribution of the Program outcome Borro Factors other than RCIP likely Relation to RCIP wer to have contributed to Count achievement of the PDO ries Malaw • Technology improvements • The RCIP was the instrument used by the Government to i in the sector; incentivize private operators to invest in connectivity • Private sector investment infrastructure in Malawi. • The initiative allowed Malawi to access to high-speed internet and improved technology brought along by the submarine cables. Moza • Technology improvements • The RCIP kick-started a series of regulatory reforms enabling mbiqu in the sector; environment for a third operator to enter the market, bringing e • Private sector investment more private sector investment in ICT. • Competition resulted in all the operators investing to upgrade their technology and offer better services to citizens. Tanza • Technology improvements • The RCIP is listed as one key initiative under the National ICT nia in the sector; Infrastructure Development Program, envisaged from the start • Private sector investment; as complementary to the NICTBB. • National ICT Broadband • The RCIP support was instrumental for the NICTBB to be built Backbone (NICTBB) (financed in accordance with technical specifications, and to be with a loan from China); operational early on (RCIP financed field supervision of the • Performance Results and contractor, as well as direct technical assistance for its Accountability Project (PRAP) operationalization according to the open-access principle to (WB Project) ensure affordability). Page 17 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) • The RCIP and PRAP coordinated and shared responsibilities (PRAP worked on the development of the eGov Agency, and an eGov strategy, while RCIP focused on connectivity for government institutions, and development of eGov applications. Overall Efficacy Rating Substantial 5. Efficiency (i) Operational and administrative efficiency Efficiency performance under this heading was mixed. The project was closed on June 30, 2016, 16 months after the original closing date. Overall, delays were contained, given the difficulties of administering a project implemented in three different countries. Similarly, management costs were contained in the case of Tanzania and Mozambique. In Tanzania, project management costs are equal to 4 percent of total project costs, a much lower proportion than the one observed in comparator countries, suggesting that the project benefitted from an efficient implementation. In Mozambique, management costs accrued to 10 percent of total project costs. This figure is slightly higher than the 8 percent average observed among the RCIP projects used as comparators but does not indicate major inefficiencies. On the contrary, project management costs in Malawi were 23 percent of total project costs, compared to an average of 8 percent observed among comparable connectivity projects in East and West Africa, due to the set-up of a stand-alone Project Implementation Unit (PIU), rather than Government-paid public servants to implement the project and large overhead costs, including rental expenses, and per-diems for public servants receiving training. (ii) Economic Analysis An economic analysis of the program's impact in each beneficiary country was conducted at appraisal and at closing. However, it should be noted that there were real difficulties in assessing efficiency in a project like this. For instance, the introduction of telemedicine has brought additional benefits that could not be easily quantified. Developments presented in the ICR and related to program efficiency in each country are summarized below: Tanzania Page 18 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) At appraisal, quantifying economic and financial impact of the project was challenging, but the following benefits were identified for key components: • Under the connectivity component, expected benefits included: (i) 14 million new mobile subscribers and 16 million Internet users over 10 years, bringing the total teledensity up to 52 percent by 2019, from about a baseline of 22 percent; (ii) it was estimated that 23 thousand new public sector users will get access to the internet services over the next 10 years, as a result of the expanding Government network and internet connection to public institutions at a district level; (iii) expanded access to the ICT services in rural areas; (iv) the substitution of transportation by phone calls and mobile applications resulted in savings estimated to be close to 0.11 percent of GDP in ten years (2018); (v) the impact of productivity income increase through the use of mobile phones in the agriculture sector was estimated to reach 0.1 percent of GDP in ten years, and (vi) other economic benefits not covered in financial analysis include access to health services, social inclusion, and new services. • Under the transparency component, expected economic and financial impact was challenging to assess as much of the data was to be generated during the project to create a baseline to measure the economic effects covering variables as improvements in terms of cost, quality, time or productivity, human capital, and technology. However, the eGovernment applications were to be a major driver for an increase in economic growth, innovation in the ICT sector, and international competitiveness. At program closing, the ICR identified the following benefits : • The increase in broadband penetration (from 42 percent in 2010 to 69 percent in 2017) is estimated to contribute to GDP in the range of US$1,186 to US$1,483 million, or from 1.6 percent to 2 percent of total GDP created in Tanzania during the period 2011-2020. Considering the NICTBB investment cost and the cost of field supervision and technical assistance financed under RCIP, the IRR on this investment is estimated to range between 27 percent and 32 percent, significantly higher than the prevailing interest rate in Tanzania (12 percent). • The average annual expenditure of a Tanzanian household on public services is estimated to decrease from US$8.69 in 2017 to US$4.18 in 2020, resulting at the country level in savings in household expenditure equal to US$122 million over the period of 2017-2020. Mozambique • Under the connectivity component, expected benefits at appraisal were: (i) an increase in new mobile subscribers and new Internet users that was to shift total teledensity (fixed and mobile) up to 32 percent by 2019, from a baseline of about 20 percent, (ii) public sector users would account for approximately 8-10 percent of total internet bandwidth, and (iii) new public sector users over the next 10 years are expected to amount to around sixty thousand, and (iv) the program economic rate of return was assessed to be positive, with an estimated net present value of economic benefits equal to US$205 million. Other non Page 19 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) quantified benefits include travel cost and savings, improved public access to information and services, access to education, health, social cohesion and new services. • Under the transparency component, as in the case of Tanzania, expected economic and financial impact was challenging because of insufficient data in Mozambique and much of the data was to be generated during the project to create a baseline to measure the economic effects. At program closing, the ICR identified the following benefits: • The increased competition in the ICT market allowed an additional 9 million Mozambicans to gain access to mobile broadband. Associated economic benefits are estimated to range between US$296 and US$370 million, or 2.1 percent and 2.7 percent of total GDP created in Mozambique during the period 2012- 2019. Mobile broadband penetration increased by 9 percentage points in 2012 with respect to the previous year, 15 percentage points in 2013, and 7 percentage points in 2014. • Economic benefits deriving from the introduction of eGovernment applications are difficult to quantify, but the average annual expenditure of a Mozambican household on public services is expected to decrease from US$5.70 in 2017 to US$2.74 in 2020. At the country level, this would result in savings in household expenditure equal to US$42 million over the period 2017-2020. Malawi • At appraisal, it was anticipated that broadband penetration will rise from a rate close to zero to approximately 1.3 percent over a ten year period, equivalent to over 100,000 broadband users. Indirect benefits include: (i) a contribution of broadband services to GDP in Malawi of around 0.7 percent by 2014, in addition to the expected contribution of mobile services of around 4 percent, (ii) the contribution of the broadband segment market in Malawi was estimated at approximately US$86 million in tax revenues to the Government over a ten-year period; (iii) the provision and consumption of broadband services was to generate large consumer benefits, and (iv) a positive indirect impact on the economy through information flows & market efficiency, innovation, local content stimulation, travel cost & time reduction, and new services. At program closing, the ICR identified the following benefits: • The project is estimated to have created value equal to about US$82 to US$103 million over ten years, or 0.7 percent to 0.9 percent of total Malawi GDP during 2016-2025. Discounted at 16 percent, the Net Present Value (NPV) of the expected contribution ranges between US$62 and US$77 million. Considering both public and private costs incurred in the PPP, the internal rate of return (IRR) on this investment is estimated to range between 30 percent and 47 percent, much above the 16 percent discount rate. Page 20 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 79.00 ICR Estimate  25.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the RCIP3's objectives is high, and both the efficacy toward the three program's objectives and the program's efficacy are substantial. Moreover, 16 out of the 17 PDO targets were either met or surpassed. An additional note from the Bank team, which is consistent with the ICR content, supports the view that attribution of the achieved outcome should accrue to the activities funded by the program, but also to ongoing or parallel activities (for instance China support to Tanzania) in the ICT sector in the three countries, including the role played by private sector investments and technological innovation. Based on the above program review, the program outcome is rated as Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Overall, the risk to development outcome in the three countries is low, as ICT upgrade is among the priorities in the three beneficiary countries. In Tanzania, digital development is at the top of the country’s Development Vision 2025 and the Tanzania National Development Strategy 2016-2020. The Government and the Bank are currently preparing a follow-up operation, (Digital Tanzania) which aims to build on program achievements through notably strengthening the ICT policy and promoting broadband for rural communities. In Mozambique, ICT is a priority area in the National Financial Inclusion Strategy 2016-2022, the ICT market liberalization process is continuing, and the Government is further strengthening the ICT and eGovernment Page 21 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) policy environment. In Malawi, ICT is a key priority area for Malawi’s development both in its Vision 2020 and the Malawi Growth and Development Strategy III. The Government and the Bank plan to build on achieved outcomes by investing in the follow-up Digital Malawi Program. The currently active Malawi Digital Foundations Project aims, among other things, to further strengthen the policy and regulatory framework, promote development of digital skills, and improve access to high-speed broadband connectivity to government, citizens, and businesses across the country. 8. Assessment of Bank Performance a. Quality-at-Entry The project design was informed by a comprehensive sector policy dialogue between the World Bank team and the Governments of beneficiary countries; it relied on a large body of analytical work, and the program's regional approach promoted a synchronized action on ICT development in the sub-region. The RCIP was designed as a modular program, whereby countries could join the program as and when they become interested and ready, and this seems to be a very pragmatic approach. The project appraisal also drew on the lessons learned from the Quality Assurance Group (QAG) of the RCIP APL 1. However, implementation arrangements were designed to fit the country-specific requirements, and not to match the requirements of the regional integration program. That is why the three country-specific operations were implemented as stand-alone projects. In Tanzania, a Project Preparation Advance (PPA) helped to expedite the conditions for project effectiveness, and the Bank team engaged in a comprehensive dialogue with national stakeholders, before designing the project and leveraging existing governance structures and procedures. The Tanzania project was designed to be complementary to other ICT initiatives active in Tanzania at the time, and its scope was limited to the areas where the Government needed more technical assistance to make the overall National ICT Infrastructure Development Program successful. Early coordination and complementarity were also ensured through a World Bank s public sector modernization project supporting an eGovernment strategy and an Executive Agency. Potential risks were adequately identified and mitigation measures were adequate and effective, but not sufficient for the specific risks related to procurement. In Mozambique, a PPA supported a comprehensive list of preliminary studies and workshops to discuss sector issues and strategy, and enabled a wide participation of various institutions in the preparation of the project led by the Ministry of Science and Technology (MST), in coordination with the Ministry of Transport and Communications (MTC). A PIU was put in place, staff were timely hired and the PIU office were operational by project approval. In Malawi, the Bank supported development objectives that were strategic to the Government. Implementation arrangements drew from achievements from the World Bank's Business Page 22 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) Environment Strengthening Technical Assistance Project (BESTAP), with responsibilities well spelled out for each contributing agency. Extensive consultations with stakeholders allowed the latter to have inputs into project design, and an inter-ministerial Project Steering Committee was set up to oversee the project. Potential risks were adequately identified at preparation and the suggested mitigation measures undertaken were effective. As developed under Section 10, An Environmental and Social Management Framework (ESMF) and a Resettlement Policy Framework (RPF) for each country were prepared and approved by the World Bank, and specific arrangements were detailed in the Technical Annex for each country. Similarly, strengths and weaknesses of the existing fiduciary systems in each country were discussed in the Technical Annex, including adequate arrangements to ensure smooth procurement and financial management of respective country projects. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Supervision in Tanzania was entrusted to a Bank team comprised of both connectivity and digital government experts that could offer technical support on these two different areas of implementation. The two restructurings in 2015 and 2016 responded to the implementation opportunities, challenges and ground changing circumstances, as well as progress against achievement of targets. Some shortcomings included the change in TTLs which caused delays in implementation support, and irregularity of supervision missions during the transition period, which led to shortcomings in both implementation support and the reporting of progress information. Finally, there was limited coordination with the World Bank team supervising the PRAP project, whose activities were central to the success of the eGovernment component. In Mozambique, project supervision was facilitated by the extensive support of the Bank team and a country-based TTL for most of the project implementation, allowing extensive technical support and increased coordination with the different Government actors and other stakeholders. The team took advantage of the restructuring opportunities to revise some PDO and intermediate indicator targets toward strengthening the relevance of the results framework. Shortcomings included delays in completing the legislative process of supported ICT-related reforms, which were approved only after the project's close. In Malawi, the Bank team offered valuable technical assistance to ensure that the new e-legislation benefitted from the best international practice. The project restructuring was timely, and implementation reporting was candid and timely. Moderate shortcomings in Bank supervision include: (i) the delays in receiving the required procurement approvals from the Bank related to bid evaluation and contract award to SimbaNET, (ii) the significant overrun in project management costs due to limited oversight by the Bank Page 23 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) team, and finally, (iii) a missed an opportunity to update targets in the M&E framework to reflect results observed during the 2016 restructuring. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The program design used industry standards to set the PDO performance indicators as detailed in Annex 2 and Table 6 of the country-specific Technical Annexes. The country-specific result framework allowed information to be aggregated across countries in order to report on progress at the regional level. However, this regional harmonization of the M&E framework did not allow the definition of PDO indicators to fully capture results derived from an integration program at the sub-regional level. The result framework was customized to the country-specific operations, and not to the results to be achieved by the program at the sub-regional level. While the results framework was adequate to monitor and report on country-level performance, it was not an appropriate instrument to gauge the value-added provided by the program at the sub-regionaln level. This was difficult, because the three countries invested in different activities, pursued different implementation strategies, and their implementation timelines did not coincide. b. M&E Implementation In Tanzania, data was collected by multiple institutions including the President’s Office, the Ministry of Works, Transport and Communication according to their respective mandates and responsibilities. Capacity building was provided to these institutions to enhance their M&E expertise, and appropriate data was timely collected and communicated to the Bank team. Few intermediary indicators were deleted following the cancellation of the land records component. In Mozambique, data was collected by a dedicated M&E specialist within the PIU from different sources, and data collection was difficult, because it was not part of the traditional information monitored by the regulator. In Malawi, data collection was spotty during the first part of project implementation, but was tightened up after the medium-term review (MTR). The sector regulator (MACRA) Page 24 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) was the primary agency to carry out M&E, as project indicators were already being collected as part of its regular sector monitoring. An MTR was completed in June 2012 and a few indicators were dropped at restructuring as they were difficult to monitor. Overall, M&E was conducted through supervision and MTR missions, and detailed aide-memoires and Implementation status and results rports ( ISRs) reporting on the status of PDO and intermediate indicators were instrumental in providing a realistic picture of the program's progress toward the expected outcomes. c. M&E Utilization M&E data and information (especially through ISRs) were useful to keep all stakeholders abreast of the progress made in the implementation of country-specific projects. That information was used by TTLs, in cooperation with PIU and respective government offcials, to initiate implementation adjustments using second level restructurings, including the revision of PDO and intermediate indicators, budget reallocations, and catch up with accumulated delays. Those data were also used in the preparation of the country's strategic documents. For instance in Malawi, the PDO and outcome indicators were used by the Government as one way to monitor the implementation of ICT-related objectives in the Malawi Growth and Development Strategy II. M&E Quality Rating Modest 10. Other Issues a. Safeguards Regarding safeguards compliance, the program and country-specific projects were assigned the B category at appraisal, and only two safeguards policies were triggered as follows: (i) OP 4.01 (Environmental Assessment), and (ii) OP 4.12 (Involuntary Resettlement), to take care of land acquisition for terrestrial facilities. An Environmental and Social Management Framework (ESMF) and a Resettlement Policy Framework (RPF) for each country were prepared, and approved by the World Bank. In the case of Tanzania at project completion, an Environmental and Social Safeguards audit confirmed that the construction of ICT networks and infrastructure in rural connectivity sites complied with the environmental and social requirements outlined in the ESMF and RPF. In the case of Mozambique, the main works identified as risk-triggering were related to the remodeling of existing infrastructure to accommodate servers and networks in public/government buildings or facilities. At project closure, no environmental or social issues were reported. Page 25 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) In the case of Malawi, an Environmental Action Plan (EAP) was prepared in 2014 for the deployment of the fiber-optic link, and it was approved by the Bank and the Government. At program completion, the Environmental and Social Safeguards audit found that the projects complied with the environmental and social requirements outlined in the ESMF, the RFP and the EAP. b. Fiduciary Compliance Financial management: An audit prepared at the program's closure concluded that (a) the Annual Work Plan and Budget was prepared in accordance with the Financial Procedures Manual and was successfully revised by the team, (b) annual audits were performed on a regular basis, in compliance with the World Bank’s rules, (c) the team submitted Interim Financial Reports on time in accordance with the standards, (d) the audit was conducted annually by qualified external auditors, and (e) no complaints about the violation of the Governance principles nor any act of corruption were raised at any point. Procurement: The ICR did not provide an overall assessment of the procurement activities during projects' implementation. However, ISRs and Aide-memoires were specific on challenges met and solutions initiated by the team in the acquisition of goods and services planned to be procured under the country's specific projects. No mis-procurement was reported, although most implementation delays arose from the delays in processing bids, and the Bank team's overall assessment of the procurement function is Satisfactory. c. Unintended impacts (Positive or Negative) The ICR indicated on p.20-21 that the program contributed to the creation of an enabling environment to substantial participation of the private sector in all three countries. Similarly, the supply of broadband to the rural sector, and the development of eGovernment applications have permitted underserved rural areas to benefit from Government services, leading to a certain level of poverty reduction and shared prosperity. d. Other The ICR noted (p.21) that the telemedicine in Tanzania has created positive spillovers of knowledge exchange between doctors and technicians from the reference hospital and those from the local hospitals. Page 26 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Moderately Moderately Bank Performance --- Satisfactory Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR identified a number of lessons and recommendations that are rephrased and summarized below: (i) Efficacy of WBG's regional programs and projects requires well-devised implementation arrangements: While there was value added for the RCIP to promote a coherent action on ICT investment in the sub-region, implementing three different projects under one World Bank code was constrained by lack of operational flexibility, and an insufficient budget for their preparation, supervision and reporting. In order to minimize the above drawbacks while maintaining the advantages of a regional approach, it is advisable to develop a customizable regional umbrella program for digital development in a specific region but ensure that each country's project is implemented as a stand-alone operation. (ii) Because of rapid change in the ICT sector, a sequential approach might result in a better project design. A multi-phased approach allows more flexibility in adapting the project’s design to changing circumstances in the ICT sector and in the country's context. Therefore, a clear sequencing of activities at the preparation stage, using a multi-phase approach enables smoother implementation and strengthens the project’s relevance throughout its course. The identification of this sequencing already at the preparation stage might have facilitated a more efficient and organized implementation. (iii) As specific eGovernment applications tend become obsolete quickly, the Bank should aim to support activities to the foundations of digital government: The Bank might want to consider focusing its investments in building the foundations of digital government (interoperability modules and Application Programming Interfaces (APIs), cloud technology, cybersecurity or data warehouses) instead of developing eGovernment applications, because the latter might become obsolete very quickly. Moreover, developing eGovernment applications requires strong coordination between different branches of government, thus involving significant coordination risks beyond what the Bank is prepared for. (iv) When designed appropriately, PPP arrangements are effective models to support ICT projects and programs: The positive outcomes from the RCIP3 suggest that the PPP models are both effective in delivering results, and efficient in their cost structure and organization, and could be adopted in future connectivity operations. In Malawi and Mozambique, the acquisition of wholesale bandwidth by the government created the incentives and reduced investment risk for private operators to build out new backbone infrastructure which Page 27 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RCIP - Phase 3 (P111432) could serve both government and the private market. In Tanzania, the reverse subsidy auction model adopted for rural mobile connectivity ensured the highest value for money and maximum leveraging of private financing through the competitive award process while shifting all financial and operational risk for the mobile sites and services offerings to the private sector operators. 13. Assessment Recommended? Yes Please explain A PPAR assessing the performance of the RCIP3, and the entire RCIP would allow to learn lessons accruing to the integration experience, which was not the focus of this ICR. 14. Comments on Quality of ICR The ICR is comprehensive and focused on results achieved in the three beneficiary countries, and the ratings are in line with the Bank's guidelines. The theory of change underpinning the program and recontrcuted in the ICR was well thought out, and the lessons learned are sound and derived from the program's implementation experience. Areas that could have been improved are as follows: (i) the assessment approach needed to be more focused on the program and project objectives, and not on the beneficiary countries; (ii) the regional integration dimension of the program and the lessons learned from its implementation should have been discussed further; (iii) because of a the role of Governments' policies and the China's loan to Tanzania, and that of the private investment in the ICT sector and the technology innovation in all countries, the ICR should have devoted further analysis to the outcome attribution issue, and finally (iv) the table presenting key outputs by component (Table I. B) is critical for the assessment of the program efficacy and needed to be prepared and presented more carefully. Upon request, the program team provided an improved table on outputs by components, and additional information explaining better the program outcome attribution. a. Quality of ICR Rating Substantial Page 28 of 28