ICRR 13028 Report Number : ICRR13028 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 05/04/2009 PROJ ID : P075808 Appraisal Actual Project Name : Alpha Maroc Project US$M ): Project Costs (US$M): 5.36 3.10 Country : Morocco Loan/ US$M ): Loan /Credit (US$M): 4.10 1.20 Sector Board : ED Cofinancing (US$M ): US$M): 0.00 0.00 Sector (s): Adult literacy/non-formal education (80%) Central government administration (20%) Theme (s): Education for all (67% - P) Gender (33% - S) L/C Number : L4679 Board Approval Date : 09/18/2002 Partners involved : Closing Date : 06/30/2006 12/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Manisha J. Modi Ridley Nelson Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: According to the Loan Agreement and the statement on the second page of the PAD, “The The objective of the Project is to test new approaches to teaching quality and literacy program management that will enable the Borrower to implement nation -wide programs that result in lasting literacy outcomes .� This objective is consistent with the intended use of the instrument, a Learning and Innovation Loan . Note different objective statements were used in some sections of the PAD. Annex 1 of the PAD states that the PDO is “to offer Note: [Note the poorest adults an opportunity to obtain basic education through high quality, well managed literacy programs� (p. 12), while Annex 2 of the PAD states that “The development objective of the project is to reduce the level of illiteracy in the poorest sectors of the working adult population, particularly among rural women and girls.� (p. 14) ] b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): US$ 3.79 million at appraisal, US$ 1.48 million actual (39%)) Improving Program Quality (US$3 39 %)) The first component aimed to improve the quality of literacy programs . It prioritized stakeholder participation, potential sustainability and cost-effectiveness in designing interventions . The first sub-component was revision of curriculum and teaching materials, including analysis of literacy curricula needs, revision /re-printing of existing literacy curricula, design of new specialized literacy curricula and the creation of a document collection /library to be made available to Literacy Department (DLCA) personnel, researchers, and service providers . The second sub-component was strengthening teaching capacity of service providers, including training literacy trainers and senior trainers for educational reform and professional research and communication . Strengthening Management of the System (US$0 236 %)) The US$ 0.76 million at appraisal, US$ 1.79 million actual (236%)) second component aimed to enhance the capacity of all levels of the system to effectively manage large -scale programs, in the context of a deconcentration movement within the Ministry of Labor, Professional Training, Social Development and Solidarity. This component had two subcomponents . The first was to strengthen administrative management capacity through developing and implementing a management plan for the deconcentration effort, training for literacy project management, and monitoring and evaluation . The second sub-component was to strengthen local operations through literacy assessment training, development of post -literacy strategies, and financing innovation to improve program quality . Regular evaluation and analysis of literacy programs (US$0 25 %)) US$ 0.77 million at appraisal, US$ 0.19 million actual (25%)) , with a view to preparing to take successful innovations to scale . (This activity was not considered a formal component but was costed separately in the cost tables .) d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project’s Mid-Term Review (MTR) in September 2005 noted slow implementation, slow disbursement and some cost savings, and recommended a partial loan cancellation and extension of the closing date . The Loan Agreement was amended in December 2005, canceling Euros 1.5 million (US$1.8 million equivalent) and extending the closing date by 18 months, to December 31, 2007. The allocations to the Quality Improvement component and the ongoing evaluation component/activity were decreased, while the allocation to the Management Capacity Strengthening component more than doubled, to US$ 1.79 million. There is considerable confusion within the ICR regarding final project cost, total loan proceeds disbursed, and final borrower contribution. According to paragraph 19, the total cost of the 3 components ultimately came to US$3.46 million, while according to paragraph 32, by closing, the Bank had disbursed Euros 1.2 million (US$ 1.76 million equivalent); suggesting a final borrower contribution of $US 1.76 million. However, Annex 1 indicates that total project costs came to $US3.1 million, with Bank disbursements totaling US$ 1.57 million and borrower contribution totaling US$1.57 million. 3. Relevance of Objectives & Design: Based on substantial relevance of objectives and modest relevance of design, the overall relevance of the project is rated modest . Objectives The relevance of the project's objective – to test approaches to teaching quality and literacy program management to inform future nation-wide literacy programs – is substantial . The 2001 Country Assistance Strategy (CAS) noted the link between illiteracy and poverty and identified education as a priority--in particular, basic education, non -formal education and adult literacy . In addition, it recommended community involvement and deconcentration in order to reach the poorest populations, in rural and peri -urban areas. Finally, it stressed that "the Bank's intervention will seek to draw lessons from alternative literacy approaches recently developed with the private sector and the NGOs, and from the best international experiences ". The 2005 CAS, which was in effect at the project ’s closing, supports improving access to basic services, social programs, and economic opportunities in pockets of poverty and vulnerability; and improving education system efficiency . With respect to the country ’s social, economic, and public policy context, at the time of project appraisal, Morocco ’s recently adopted national education Charter targeted adult illiteracy, aiming to reduce the rate to 20% by 2010 and to eradicate it by 2015 (from an estimated rate of 50% in 1999). Also, according to the ICR, the implementation of the literacy component of the Bank ’s most recent education operation, the BAJ 1 Basic Education Project demonstrated the Ministry of Labor’s motivation to implement a literacy project . Design The relevance of the project design is modest . Alpha Maroc’s design took into account lessons learned from recently implemented national adult literacy programs in other developing countries, incorporating successful strategies such as adapting curriculum content to learner needs and utilizing effective partnerships between government and private /non-profit service providers. The project design also drew on lessons learned during the BAJ Basic Education project; namely, that instructional strategies from primary education were not effective for adult literacy and that the expansion of literacy programs meant a greater management burden at all levels of the system, and scarce resources needed to be used more effectively . Given the CAS’ suggested approach, as well as the ambitious goals of the education Charter and the modest achievements of the BAJ1 Project, it was appropriate for the design to focus on piloting and evaluating successful approaches to reducing illiteracy, while strengthening program management capacity, in order to lay the groundwork for scaling up literacy programs . However, the lack of an evaluation design was a major design flaw . Evaluation design is particularly important in a project and instrument with an explicit learning objective, and the review therefore places higher weight on this weakness . The results of the project were expected to provide the knowledge required in the areas of teaching quality and adult literacy program management. However, even at the appraisal stage, as reflected in the different objectives formulations, there seemed to be confusion about whether the project was designed to pilot and demonstrate the effectiveness of the activities implemented, or to immediately scale them up . A LIL is the ideal lending instrument for the original goal of piloting and learning from new approaches to decrease illiteracy, but should not have been used to support capacity building and quality enhancement of a national program . As the ICR noted, “It is difficult for a LIL to test new approaches and at the same time achieve major results regarding increased participation or increased performance and satisfaction during the life of the project .� (p. 5) 4. Achievement of Objectives (Efficacy): Objective : test new approaches to teaching quality and literacy program management that will enable the Borrower to implement nation -wide programs that result in lasting literacy outcomes Outputs The competency-based curriculum was created . The ICR states that it was used to revise the existing curricula . However, it also notes that due to delays in selecting and paying consultants, the competency -based curriculum was not completed until December 2007, while revision of the existing curricula was completed in 2005. Eight existing curricula were revised, falling somewhat short of the target off 11. In any case, only 60% of trainers use the revised curricula, and many felt the specialized curricula did not address sector needs . Training targets appear to have been met, but the ICR is again somewhat ambiguous on this point (see section 15 on quality of ICR). Twenty senior and 100 decentralized DLCA staff, and 200 public providers were trained on use of the teaching guide. Although DLCA staff were largely satisfied by the training on the teaching guide, coverage was low and there were concerns about the guide itself . Less than half of NGO trainers felt that the objectives of the training on andragogy, communication and monitoring were fully met . Only a third felt the M&E training objectives were fully met. The document collection was completed just before the project closed, and the newsletter was never published. The deconcentration plan was produced in early 2006 and training was provided to DLCA staff . There is no mention of the status of implementation. The target of 200 managers trained in project management were approximately met; 290 managers were trained in administrative and financial management skills, but only 100 were trained in M&E. Nearly two thirds of participants felt the training was not adequate for their needs, and DLCA staff and public providers requested more training on monitoring and evaluation . Some M&E tools were developed and system hardware was purchased . A post-literacy seminar was held in 2006 and a plan produced in 2007. Small grants were provided to 21 NGOs to carry out innovative literacy sub -projects, but none had been completed by the end of the project . Three annual reports and some studies were planned; however by the time the project closed, only two reports had been produced—in the last two months of the project and the studies were canceled during the MTR . Outcomes There were three main expected project outcomes : participation of the poorest adults in the literacy program, and performance and satisfaction level of beneficiaries, and --arguably the most important--knowledge gathered from the LIL. The project aimed at contributing to decisions regarding subsequent scale -up and to do so, the loan was to finance evaluations and determine whether : users find that curricula have been adapted to their needs; teacher training and support have been institutionalized; literacy institutions and providers show increased capacity to manage program growth; establish an effective sector evaluation and monitoring system; unit costs are considered reasonable; and political will exists to support scale -up. While many of the project's outputs were achieved, including launching the first cohorts of innovative literacy sub -projects, none of these outcome indicators was measured, let alone shown to have been achieved before literacy programs were massively scaled -up. Learner enrollment rates more than doubled between 2002-2008. However, the strongest increase in the number of people enrolled (57%) happened between 2003-2004, before project activities had gotten underway . This growth can be attributed to earlier literacy efforts and the broader environment of increased government commitment to eradicating adult illiteracy. The Borrower’s ICR notes that enrollment in literacy programs increased from 470,000 to 709,000 between 2004-2007, and that dropout rates fell from 20% to 11% after 2003. However, although the national adult illiteracy rate seems to have fallen from a baseline of approximately 50% to 43% at project's end, it is clear that this trend was underway even before the project was implemented, and there are no pre - and post-assessments of literacy program participants . Finally, the ICR states clearly that the project ’s development objective was not met : “The project was not able to test new approaches to literacy.� (p. 9) Overall, efficacy is rated modest . 5. Efficiency (not applicable to DPLs): No rate of return analysis was done, either at appraisal or project completion . One of the project’s operational objectives was to increase management and educational efficiency . The main indicator of efficiency gains was unit costs for literacy programs; yet, there were no baseline data, nor any monitoring of the evolution of unit cost during project implementation. There is some evidence to suggest that unit costs decreased between 2003-2007; estimates show costs were roughly halved, suggesting efficiency gains . However, in the absence of comparable data on participant dropout rates, satisfaction, and learning outcomes, it is not possible to evaluate whether efficiency truly increased. Given the scarcity of evidence, efficiency is rated modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on the modest ratings for relevance, efficacy, and efficiency, project outcome is rated unsatisfactory . a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: A major risk to the development outcome is that, in the rush to scale up literacy programs, the focus on learning and innovation to improve program effectiveness and efficiency will be lost, especially now that the LIL has closed . Another concern is that gains that were made could be lost . These risks are deemed moderate . The Government continues to prioritize adult literacy, and a MTEF for the literacy sub -sector has been prepared which indicates the need for additional resources to maintain even a modest increase in the number of beneficiaries in the coming years. The Government recently announced a new National Agency for Literacy, which will hopefully mobilize additional financial and human resources to continue to improve program efficiency . However, with these ambitious goals, there is a risk of inadequate funding . Finally, as of the writing of the ICR, the Government and World Bank had started discussing how the Bank ’s next education operation in Morocco, the Morocco Basic Education Reform Program II - PARSEM II (P095714) Project might build on the accomplishments and address the deficiencies of Alpha Maroc; in particular, the M&E system . In addition, the EU was preparing a Euro 17 million grant that might build on activities such as capacity building and M&E. These future projects should help ensure that gains achieved during the project will not be lost, and also limit the risk of inadequate funding for literacy activities . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: There were a number of problems with the quality at entry. The PAD demonstrated lack of clarity as to the objectives, resulting in ambiguity in the appropriate outcome indicators . This problem was compounded by the total lack of a monitoring and evaluation framework, as confirmed by a 2003 QAG QEA review, which rated quality-at-entry marginally satisfactory (equivalent to moderately unsatisfactory under the current rating scale .) The risk assessment was inadequate, failing to anticipate the institutional problems that would likely result from rapidly expanding literacy programs at the same time as the launch of a major decentralization process and the organizational restructuring of the national literacy efforts (the government doubled its enrollment targets for literacy programs, and the DLCA was transferred to the MOE Secretariat of State after Board approval but before the Loan Agreement was signed .) These developments compounded the Bank ’s over-estimation of DLCA’s management and procurement capacity . The PAD deemed procurement capacity to be satisfactory, noting the amount of technical assistance contracts would be low; however project implementation was delayed, partly by the difficulties in hiring qualified national consultants specializing in literacy . Quality of supervision was mixed. The team should have reassessed project design, implementation arrangements, and DLCA capacity in light of the new government legislation, personnel changes, and organizational restructuring. There was lack of candor in performance reporting, particularly during the first two years of the project, when implementation was seriously delayed and when action should have been taken to get the project back on track . Aides Memoire did not systematically address financial management (FM) and procurement and few missions included FM and procurement specialists, although it was clear fairly early on that difficulties with Bank procurement procedures were contributing to implementation delays . Toward the latter half of the project, a Bank procurement officer joined the country office, and a consultant was eventually hired to provide technical assistance on monitoring and evaluation and other issues . Two years into implementation, the project’s design was finally examined, future scenarios considered, and discussions initiated with the Government, which ultimately influenced the recommendations of the mid -term review. There was a high level of turnover in project leadership, but the last two TTLs were based in the country office and seemed to bring some much-needed realism and leadership. at -Entry :Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Unsatisfactory 9. Assessment of Borrower Performance: The Government demonstrated a high level of commitment to increasing adult literacy, but less commitment to the project’s development objective. The national policy and administrative infrastructure with respect to education and adult literacy dramatically changed between project approval and effectiveness, with little analysis of the potential consequences for the implementation of Alpha Maroc . For example, although the creation of a new Secretariat of State for Literacy and Non -formal Education (SEANF) within the Ministry of Education was good for raising the profile of the literacy program and mobilizing resources, transferring the DLCA to SEANF created problems for the implementation of Alpha Maroc . The new Secretary of State prioritized increasing enrollment in the adult literacy programs over improving quality and efficiency, and had a much broader agenda of deconcentration and reorganization . The SEANF did not have its own budget within the MOE, and there was a difficult relationship between the first Secretary of State and the Minister of Education. In addition, despite the new responsibilities, SEANF budget remained flat during the project ’s implementation, and DLCA had insufficient personnel and lacked logistical resources especially at the local level . The implementing agency, the DLCA, suffered from inadequate technical and procurement capacity . While the inadequate capacity was partly caused by low staff morale and commitment, high staff turnover, and difficulty recruiting and keeping highly skilled personnel, the DLCA took little initiative to resolve the problems, even when the Bank team provided specific recommendations . In addition, the deconcentration process was implemented without clarifying regional and provincial level personnel's responsibilities until an organizational chart was created, in the last year of the project . a. Government Performance :Unsatisfactory b. Implementing Agency Performance :Unsatisfactory c. Overall Borrower Performance :Unsatisfactory 10. M&E Design, Implementation, & Utilization: Overall, M&E quality is rated negligible. Design: There was no planned evaluation . In fact, despite the newly-initiated deconcentration effort, the project did not define the respective roles and responsibilities of the central and local levels, nor even include specific activities to develop the virtually non-existent M&E system. Implementation: Lack of resources made it difficult for DLCA staff to distribute M&E tools to providers . In 2004, the Bank mobilized a Trust Fund (independent of the project) to hire a consultant to help develop an M&E Plan, including the definition of key indicators, and a training strategy . Some tools were produced and some staff were trained . The project financed key equipment (computer and printers) to support the M&E process within DLCA . Beneficiary Assessments were not carried out, but two years’ worth were produced just before project closing . All studies were canceled. Utilization: Many indicators outlined in the PAD were not monitored and not enough data were collected to assess the quality and efficiency of the program. a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Women (who have a higher illiteracy rate than men ) were able to benefit more than men from the project; 82% of all literacy programs beneficiaries between 2002/2003 and 2006/2007 were women. In addition, half of the beneficiaries were living in rural areas where poverty and illiteracy are highest . DLCA generally could not or did not comply with Bank procurement procedures . The Bank should have provided more consistent procurement technical support, but the DLCA showed little initiative in improving its performance in this area. Interim financial statements were not systematically submitted as required by the Loan Agreement, but annual audits were regularly carried out . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory There were major shortcomings in the Satisfactory achievement of the main objective, which was to test new approaches to teaching quality and literacy program management. Relevance, Efficacy and Efficiency were all modest. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Unsatisfactory Quality at entry was unsatisfactory and Unsatisfactory quality of supervision was moderately unsatisfactory. A project with a learning objective should place great weight on getting the evaluation design in place at the outset, but this project had no M&E framework and no plans to create one. Borrower Performance : Moderately Unsatisfactory The Government's performance was Unsatisfactory unsatisfactory as it did not demonstrate commitment to the piloting and learning objective of the project. The implementing agency's performance was unsatisfactory; it had inadequate capacity and showed little initiative in responding to specific problems . Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. A LIL should be used to test, implement and evaluate new approaches, not to scale up existing programs . 2. An explicit evaluation design is a prerequisite for a successful LIL . 3. If successful implementation depends on consultants, a market analysis should be undertaken before approval to ensure that consultants will be available . 4. The results chain should be based on a clear understanding of the objectives and consistent translation of those objectives into the PDO language . 14. Assessment Recommended? Yes No Why? It is unsatisfactory that a LIL could be approved and made effective without any baseline data, clear results chain, or even an M&E infrastructure . A detailed study of the history of this project, from pre -appraisal to completion could shed light on how the LIL lending instrument is being used . 15. Comments on Quality of ICR: The ICR is rated unsatisfactory on balance, although some aspects were satisfactory . Although it did make a point of identifying the original project development objective, it did not sufficiently weight the objective of testing new approaches in the final analysis . The report was concise and on some aspects, the analysis was sound . However, key data were missing or inconsistent. There is considerable confusion within the ICR regarding final project cost, total loan proceeds disbursed, and final borrower contribution . According to paragraph 19, the total cost of the 3 components ultimately came to US$3.46 million, while according to paragraph 32, by closing, the Bank had disbursed Euros 1.2 million (US$1.76 million equivalent); suggesting a final borrower contribution of $US 1.76 million. However, Annex 1 indicates that total project costs came to $US 3.1 million, with Bank disbursements totaling US$ 1.57 million and borrower contribution totaling US$1.57 million. There are inconsistencies with respect to project outputs as well . The ICR states that the competency -based curriculum was created and that it was used to revise the existing curricula . However, it also states that revision of the existing curricula was completed in 2005, while the competency-based curriculum was not competed until December 2007. Section F reports that the target of trainers trained was 2,000 and that 2,400 were trained, but Annex 2 reports that the target was 2,400 and only 1,800 were trained between mid-2005 and in June 2006.) According to section F of the ICR, 94 of the targeted 160 personnel were trained in literacy assessment; however, Annex 2 reports this training was not done at all, because the cost was “deemed too high for expected benefits ." a.Quality of ICR Rating : Unsatisfactory