ICRR 13146 Report Number : ICRR13146 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/24/2009 PROJ ID : P076159 Appraisal Actual Project Name : Development US$M ): Project Costs (US$M): 3.5 3 Learning Center Project Country : Burkina Faso Loan/ US$M): Loan /Credit (US$M): 2.3 2.3 Sector Board : ED Cofinancing (US$M): US$M ): Sector (s): Telecommunications (40%) Adult literacy/non-formal education (30%) Sub-national government administration (15%) Central government administration (15%) Theme (s): Education for the knowledge economy (40% - P) Technology diffusion (40% - P) Corporate governance (20% - S) L/C Number : C3707 Board Approval Date : 07/29/2002 Partners involved : Closing Date : 12/31/2006 10/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Helen Abadzi Kris Hallberg Soniya Carvalho IEGSG 2. Project Objectives and Components: a. Objectives: The project was to test the viability of a distance learning institution in Burkina Faso and to test the ability of this approach to access international knowledge to : (a) improve the implementation of (i) the PRSP and/or the PRSC and (ii) the IDA-financed projects; and (b) help coordinate all the local training institutions in regard to the national capacity building policy. The Development Credit Agreement expressed the objectives more broadly : The objectives of the project are to test the effectiveness and sustainability of a distance learning center as part of a global knowledge -sharing network to strengthen the capacity of public, private and civil society decision -makers and implementers to design, plan and manage economic and social development policies . The PAD definition is used because it defines actions more specifically . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) Establishment of a distance learning center (US$1.2 M at appraisal, US$1.6 M actual, financed through a Project Preparation Facility --PPF--) for construction and equipment of the building by the Ministry of Economic Development. The center was a nonprofit organization managed by a board of representatives from various ministries, academia, and NGOs that were to guide development policy . The learning center was ultimately transferred to the University of Ouagadougou to operate under the Ministry of Secondary and Higher Education . (b) Support for the distance learning center (US$1.0 M at appraisal, US$0.60 M actual) during three years at a sliding rate of 80%, 60%, and 40% of operating costs. (c) Monitoring and evaluation (US$0.10 M at appraisal, US$0.10 M actual) for the evaluation of project impacts . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: This was a Learning and Innovation Loan (LIL) that was to advance in Burkina Faso the concept of the Global Development Learning Network (GDLN). The project benefitted from a PPF that was used during implementation rather than before, for component (a). Thus, the actual credit was used for components (b) and (c). After extensions totaling 22 months due to implementation delays, the project closed on October 31, 2008 and disbursed fully. 3. Relevance of Objectives & Design: The relevance of the objectives is rated high . The need to invest in the Global Development Learning Network (GDLN) was not explicitly identified in the 2000 Country Assistance Strategy objectives (doc. no. 21285), but it was consistent with its objectives . The subsequent PRSP and CAS include explicit references to capacity building needs and use of technology to deliver cost -efficient training and capacity development activities . The relevance of the design is rated modest . The project was designed on the basis of a model developed during the GDLN Program Pilot Phase (2000-2002) and incorporated lessons from the experiences of Senegal and Mauritania. The conceptual and technical aspects of bringing knowledge to low -income countries through distance learning, were relevant, but a rapid cost recovery rates were projected without a clear strategy or mechanisms to achieve financial sustainability for the learning center . There was no allowance for implementation delays . It was also assumed without evidence that a non -profit independent organization to manage the learning center would be more active than a public entity. However, cost-recovery and delays have been a problem worldwide, and no African GDLN center has been able to reach financial sustainability within three years . Furthermore, the project had a poorly developed M&E system, which is crucial for a LIL . 4. Achievement of Objectives (Efficacy): Objectives were modestly achieved . The ICR does not provide evidence on the "testing" aspect of the first and second objectives, largely because of the limited M&E system . Virtually all of the performance indicators measure outputs rather than outcomes . (a) test the viability of the distance learning center in Burkina Faso (modest ) A building was constructed in a new development of Ouagadougou; the construction was of high quality and the equipment was perfectly adequate to carry out the distance learning mission . The design of the building benefited from the experience of the other GDLN centers . However, the construction was delayed and cost more than expected, so less money and time was available for the execution of the other components . The initial design of the project planned at least 3 full years of operations before achieving cost recovery . The delay in component 1 implementation, as well as the cost overrun of the construction, led to a reduction in time and budget for the execution of components 2 and 3. At the end of the project , the center did not break even in terms of cost recovery, as projected At project end, only 43% of the expenses were covered by fees . (Target was 100%). The GDLN provides training at a lower cost (US$7000 per course) which is shared by three countries . It costs less to train through this mode than hiring outside vendors or sending participants abroad, and the fees charged to participants are a fraction of the price (ICR p. 8). The distance learning center may be a cost efficient way to build capacity – provided that training quality is comparable to alternatives, such as study abroad . (b) Test the ability of this approach to access international knowledge to improve the implementation of (i) the PRSP and /or the PRSC and the IDA -financed projects (modest ) Many activities took place, but most numerical targets were not attained . The Center has hosted 256 of PRSP-related capacity building events . (Target was 440). About 9-14 topics were covered every year (Target was 5); 116 activities were relevant to PRSP priorities (Target was 60). Although the project has exceeded the target values on the last two aspects, the number of participants fell below targets (3796 vs. 6000). The courses received favorable “level 1� evaluations and, according to the ICR led to the dissemination of good practice and South -South knowledge exchange . 96 managers of projects and programs received training at the center (Target was 140). About 3796 public and private professionals were trained by the fourth year of implementation (target was at lest 3500). 106 initiatives were created as a result of GDLN sessions related to PRSP implementation (Target was 150), but (eg. Health insurance, agriculture and fishing ) it is unclear what they consist of . (c) Test the ability of this approach to help coordinate all the local training institutions in regard to the national capacity building policy (negligible ). The project suffered from considerable turnover of important decision -makers in the ministries which were to benefit from it, as well as turnover at the implementation unit level . All local training institutions participating in the GDLN board were expected to have used the distance learning center, but fewer than 10% did so. The board did not function as expected; the members had limited commitment to the concept, and remained uninvolved in the operation of the center. There was limited policy development and effort to help the center becoming self -sustaining. To prevent closure, at the end of the project, the learning center was transferred to the University of Ouagadougou, where it receives financing from an ongoing Bank -financed post-primary education project . 5. Efficiency (not applicable to DPLs): Efficiency of project funds use is rated modest . The building cost more than expected, and its construction delays affected the delivery of the program . The learning events generated lower fees than expected . Nevertheless, the cost recovery achieved despite various obstacles suggests that the learning center may have higher efficiency indicators in the future. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance was rated substantial, efficacy was rated modest, and efficiency was also rated modest . Overall outcome is rated moderately satisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development effectiveness is rated significant . The learning center would have closed after the end of the project had it not been transferred to the University of Ouagadougou . Three provincial learning centers are expected to become operational . Given its relatively low cost recovery rate, its long -term viability of the learning center is uncertain without external financing . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: The quality at entry is rated unsatisfactory . The assumptions made in the PAD were mostly based on lessons learned in countries with a business environment that made better use of externally provided knowledge . Little consultation took place with other donors during project preparation, and capacity to manage the center was overestimated. An M&E system was not fully developed . Quality of supervision is rated satisfactory . The task team was pro-active throughout the project implementation, agreeing on strong conditions, actions and results when the credit closing extension was approved, and carried out extensive consultations with respect to the continuity of the project . Overall Bank performance is rated moderately unsatisfactory because the project outcome was in the unsatisfactory range . at -Entry :Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Overall borrower performance is rated unsatisfactory . Government performance is rated moderately unsatisfactory; although there was initial commitment to the project, most of the ministries were not allocated sufficient funds for training and capacity building that would keep the learning center functioning and viable . Training agreements with the Ministries of Economics and Finance were proposed but never finalized . The performance of the implementing agency was also unsatisfactory . The project implementation faced several difficulties. It suffered from considerable turnover of important decision -makers in the ministries which were to benefit from it, as well as turnover at the implementation unit level . The membership of the guiding board was not viable; there was limited interest in marketing, there were interpersonal problems among the staff, and also a limited knowledge of English. The manager of the project coordination unit did not provide sufficient leadership. (During the project the majority of the board members voted against renewing the contract of the manager (9 to 5), but then they renewed it anyway .) Therefore problems with the project continued . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Unsatisfactory c. Overall Borrower Performance :Unsatisfactory 10. M&E Design, Implementation, & Utilization: Monitoring and evaluation are rated modest . Since the project was a LIL, monitoring was important . However, the project coordination unit lacked capacity to develop impact evaluation tools . Thus, the design of M&E reports and data collection mechanisms were not in place at effectiveness and the project did not develop any basic M&E tools prior to the midterm review. The project used survey instruments from other operations, such as "level 1" questionnaires. Surveys were conducted and basic statistics about participants were collected, but the value added of the training that was provided through the learning center could not be established . There was limited use of the data other than utilization during supervision missions, and limited use for decision -making. Design and utilization are rated negligible, while implementation is rated modest . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): n/a. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Moderate Significant Given a relatively low cost recovery Outcome : rate, the long-term viability of the learning center is uncertain without external financing. Bank Performance : Moderately Moderately There were substantial quality at entry Satisfactory Unsatisfactory issues. Borrower Performance : Moderately Unsatisfactory The government had limited Unsatisfactory commitment, insufficient budgets for training tasks, and appointed a board that had little interest in the projec t Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The Bank has financed in various sectors innovative projects on a common design . However, country priorities and conditions differ, and design templates are not always applicable . Suitable adaptations are needed in each country in order to ensure fulfillment of the objectives . When innovative projects are implemented in low -income countries, cost recovery rates and the time needed to complete various tasks must be carefully assessed during appraisal . Overly optimistic expectations may result in lower ratings of project outcomes than more modest expectations . Project preparation funds are aimed at facilitating the preparation of crucial components so that other components can be implemented more effectively . If the funds are not used during preparation for these crucial components, delays may arise that may compromise the fulfillment of the development objectives . It is difficult to carry out innovative capacity building projects without sufficient government commitment, budgeting, and ownership by a single agency . Utilization of the project services must be ensured in advance if funds are to be used efficiently . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR was satisfactory, and frankly relates the events of the project . However, the total project cost in Annex 1 on project financing is unclear, and cost figures do not match exactly with those of the PAD . The ICR did not downgrade Bank performance to unsatisfactory given the unsatisfactory outcome and also mentioned nothing about the topic of corporate governance, which was mentioned in the PAD (p. 3). On June 16, 2009, the task manager (who was traveling) kindly provided in writing replies to questions regarding project costs, building costs, location, and other implementation -related issues. a.Quality of ICR Rating : Satisfactory