Document of The World Bank Report No: ICR0000680 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-46790) ON A LEARNING AND INNOVATION LOAN IN THE AMOUNT OF EURO 4.2 MILLION (US$ 4.1 MILLION EQUIVALENT) TO THE KINGDOM OF MOROCCO FOR AN ALPHA MAROC PROJECT August 5, 2008 Human Development Sector Maghreb Department Middle East ad North Africa Region CURRENCY EQUIVALENTS Currency Unit = Moroccan Dirham (MAD) Exchange rate (as of 9/18/2002, Board date) US$1 = Euro 0.69 US$1 = MAD 7.81 Exchange rate (as of 12/31/2007, closing) US$1 = Euro 1.03 US$1 = MAD 10.80 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS AREF: Regional Education and Training Academies [Académies régionales d'Education et de Formation] BAJ: Barnamaj Al Aoulaouaiyat Al Ijtimaiya (Social Priorities Program) CAS: Country Assistance Strategy CO: Country Office CPPR: Country Portfolio Performance Review CTF Consultant Trust Fund DH: Dirham DLCA: Literacy Department [Direction de la lutte contre I'analphabétisme] GoM: Government of Morocco ICR: Implementation Completion and Results report IIEP: International Institute for Education Planning (UNESCO) ISR: Implementation Supervision Report LIL: Learning and Innovation Loan NGO: Non-Governmental Organization PAD: Project Appraisal Document PDO: Project Development Objective M&E: Monitoring and Evaluation MoF: Ministry of Economy and Finance MoE: Ministry of National Education MEFP: Ministry of Labor, Professional Training, Social Development, and Solidarity [Ministère de l'Emploi, de la Formation Professionnelle du Développement social et de la Solidarité] MTR Mid-Term Review SEANF: Secretariat of State for Literacy and Non formal Education [Secrétariat d'Etat chargé de l'Alphabétisation et de l'Éducation Non Formelle] TL : Team Leader Vice President: Daniela Gressani Country Director: Mats Karlsson Sector Manager: Mourad Ezzine Project Team Leader: Rachidi B. Radji ICR Team Leader: Nadine Poupart IMPLEMENTATION COMPLETION REPORT KINGDOM OF MOROCCO ALPHA MAROC PROJECT (LOAN 4679-MOR) TABLE OF CONTENTS A. Basic Information...................................................................................................... iv B. Key Dates.................................................................................................................. iv C. Ratings Summary...................................................................................................... iv D. Sector and Theme Codes ........................................................................................... v E. Bank Staff................................................................................................................... v F. Results Framework Analysis...................................................................................... v G. Ratings of Project Performance in ISRs ..................................................................vii H. Restructuring (if any)..............................................................................................viii 1. Project Context, Development Objectives and Design............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 4 3. Assessment of Outcomes............................................................................................ 9 4. Assessment of Risk to Development Outcome......................................................... 12 5. Assessment of Bank and Borrower Performance ..................................................... 12 6. Lessons Learned ....................................................................................................... 15 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 16 ANNEXES: Annex 1. Project Costs and Financing.......................................................................... 17 Annex 2. Outputs by Component ................................................................................. 18 Annex 3. Economic and Financial Analysis................................................................. 22 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 23 Annex 5. Beneficiary Survey Results........................................................................... 25 Annex 6. Stakeholder Workshop Report and Results................................................... 27 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR..................... 28 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders....................... 30 Annex 9. List of Supporting Documents ...................................................................... 31 A. Basic Information Country: Morocco Project Name: Alpha Maroc Project Project ID: P075808 L/C/TF Number(s): IBRD-46790 ICR Date: 04/10/2008 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: LIL Borrower: MOROCCO Original Total USD 4.1M Disbursed Amount: USD 1.2M Commitment: Environmental Category: C Implementing Agency: Ministry of Labor, Professional Training, Social Development and Solidarity (MEFP) Cofinanciers and Other External Partners: N.A. B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 11/27/2001 Effectiveness: 05/05/2003 05/05/2003 Appraisal: 02/19/2002 Restructuring(s): Approval: 09/18/2002 Mid-term Review: 07/10/2004 09/12/2005 Closing: 06/30/2006 12/31/2007 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Moderately Unsatisfactory Government: Unsatisfactory Quality of Supervision:Moderately Implementing Moderately Unsatisfactory Agency/Agencies: Unsatisfactory Overall Bank Moderately Overall Borrower Moderately Performance: Unsatisfactory Performance: Unsatisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Performance Indicators (if any) Rating Potential Problem Project Yes Quality at Entry Moderately at any time (Yes/No): (QEA): Unsatisfactory Problem Project at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Adult literacy/non-formal education 80 80 Central government administration 20 20 Theme Code (Primary/Secondary) Education for all Primary Primary Gender Secondary Secondary E. Bank Staff Positions At ICR At Approval Vice President: Daniela Gressani Jean-Louis Sarbib Country Director: Mats Karlsson Marisa Fernandez-Palacios Marlaine Lockheed Sector Manager: Mourad Ezzine Project Team Leader: Rachidi Radji Jeffrey Waite ICR Team Leader and Nadine Poupart Primary Author: F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) According to Annex 2 of the Project Appraisal Document (PAD), the Project Development Objective (PDO) is to reduce the level of illiteracy in the poorest sectors of the working adult population, particularly among rural women and girls. To attain this objective, the following two outcomes are expected: (i) improvement in program quality owing to revision of the curricula and training of literacy teacher trainers; and (ii) strengthening management of the system through improvements in the partnership-based approach between the Government and service providers. In the section justifying the choice of a Learning and Innovation Loan (LIL) instrument (PAD, p.2), it is indicated that the project's main purpose is to test new approaches to teaching quality and adult literacy program management which, if proven effective, will enable the Government of Morocco (GOM) to implement nation-wide programs that result in lasting literacy outcomes. A project of this kind provides an opportunity to assess which approaches are best taken to scale. Revised Project Development Objectives (as approved by original approving authority) N.A. (a) PDO Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Revised Achieved at approval Target Completion or documents) Values Target Years Indicator 1 : Adult illiteracy rate (population aged 15 +) Value quantitative or 50% 40% 43% (2006; latest Qualitative) survey) Date achieved 09/12/2002 12/31/2007 12/31/2007 Comments (incl. % Completed. The 8 curricula were revised by Literacy Department (DLCA) staff. achievement) Number of trainers trained in the theory and practice of adult literacy education. Indicator 2 : Number of senior trainers trained on highly specialized issues in adult literacy education. Value quantitative or 0 2,000 trainers 2,400 trainers Qualitative) 20 senior trainers 20 senior trainers Date achieved 09/12/2002 12/31/2007 12/31/2007 Comments Number of trainers trained has surpassed target value. 41%-49% feel that the (incl. % objective of the training on andragogy (i.e., teaching to adults), communication achievement) and monitoring weree fully met (NGO). For all trainers, 34% feel that training on M&E was fully met, and 65% partially met. (b) Intermediate Outcome Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Achieved at approval Revised Completion or documents) Target Values Target Years Indicator 1 : Number of new and revised curricula designed and printed Value 0 11 8 revised and (quantitative printed and 3 new or Qualitative) curricula designed.) Date achieved 09/12/2002 12/31/2007 12/31/2007 Comments (incl. % Completed. The 8 curricula were revised by DLCA staff. achievement) Number of trainers trained in the theory and practice of adult literacy education. Indicator 2 : Number of senior trainers trained on highly specialized issues in adult literacy education. Value (quantitative 0 2,000 trainers 2,400 trainers or Qualitative) 20 senior trainers 20 senior trainers Date achieved 09/12/2002 12/31/2007 12/31/2007 Comments Number of trainers trained has surpassed target value. 41%-49% feel that the (incl. % objective of the training on andragogy (i.e., teaching to adults), communication achievement) and monitoring were fully met (NGO). For all trainers, 34% feel that training on M&E was fully met, and 65% partially met. Indicator 3 : Number of personnel trained in administrative and financial management skills and in M&E 290 trained in Value administrative and (quantitative 0 200 trained. financial or Qualitative) management skills. 100 trained in M&E. Date achieved 09/12/2002 12/31/2007 12/31/2007 Comments (incl. % Number of personnel trained has surpassed target value. But 72%-77% not achievement) consulted on training content. 62%-64% felt it is not adequate to their needs. Indicator 4 : Number of personnel trained in literacy assessment Value (quantitative 0 160 94 or Qualitative) Date achieved 09/12/2002 12/31/2007 12/31/2007 Comments (incl. % Number of personnel trained in literacy assessment lower than projected. achievement) G. Ratings of Project Performance in ISRs Actual No. Date ISR Archived DO IP Disbursements (USD millions) 1 12/23/2002 Satisfactory Satisfactory 0.00 2 03/13/2003 Satisfactory Satisfactory 0.00 3 06/30/2003 Satisfactory Satisfactory 0.05 4 12/22/2003 Satisfactory Satisfactory 0.05 5 05/11/2004 Satisfactory Satisfactory 0.05 6 09/01/2004 Satisfactory Satisfactory 0.49 7 12/23/2004 Satisfactory Satisfactory 0.49 8 04/25/2005 Moderately Satisfactory Moderately Satisfactory 0.55 9 11/15/2005 Moderately Satisfactory Satisfactory 0.55 10 06/29/2006 Moderately Satisfactory Moderately Satisfactory 0.73 11 12/20/2006 Moderately Satisfactory Moderately Satisfactory 1.05 12 06/19/2007 Moderately Satisfactory Moderately Satisfactory 1.20 13 12/27/2007 Moderately Satisfactory Moderately Satisfactory 1.20 H. Restructuring (if any) Not Applicable Disbursement Profile . 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. In 1999, the Government adopted an ambitious education "Charter" that aims to substantially increase Morocco's educational attainment. The Charter sets goals and policies across the education sector, from early childhood to adult learning. In the 2001 Country Assistance Strategy (CAS), the Bank identified education as a priority. Within the education sector, the CAS stated that the Bank's efforts would focus on non-formal education and adult literacy, along with basic education, as key areas most likely to contribute to poverty reduction. 2. As a middle-income country, Morocco registered surprisingly high levels of illiteracy. The Charter identifies the following illiteracy targets to attain by the end of the Education Decade (2010): to reduce adult illiteracy from the official estimate of 50% in 1999 to 20% by 2010, and to eradicate adult illiteracy by 2015. Considering that there were an estimated 10 million illiterate adults in Morocco, an extensive literacy training program needed to be implemented over the next 7 years. 3. In 2001 (and up until the appointment of the Government at the end of 2002), non-formal education was the responsibility of the Ministry of Education (MOE) and adult literacy the responsibility of the Ministry of Labor, Professional Training, Social Development and Solidarity (MEFP). Given the institutional capacity and the high level of motivation shown by the Labor Ministry in implementing Component D ("Improving the Effectiveness of Literacy Programs") of the BAJ1 Basic Education Project, the Bank moved forward quite quickly on developing an operation in adult literacy. 1.2 Original Project Development Objectives (PDO) and Key Indicators 4. The PDO is not consistent across the documentation. As presented in the PAD (p. 2) and the Legal Agreement, the PDO is to "test new approaches to teaching quality and adult literacy program management, which if proven effective, will enable the GOM to implement nation-wide programs that result in lasting literacy outcomes." This is the PDO against which the project is assessed in this Implementation Completion and Results (ICR) report. 5. However, as presented in Annex 1 of the Project Appraisal Document (PAD), the PDO reads as follows: "to offer the poorest adults an opportunity to obtain basic education through high- quality, well managed literacy programs". And according to Annex 2 of the PAD, the PDO is "to reduce the level of illiteracy in the poorest sectors of the working adult population, particularly among rural women and girls." This is the PDO that was used in the ISR, and against which project performance was evaluated during supervision. 6. The project had the following outcome indicators (PAD, Annex 1): - Increase in the participation of the poorest adults in efficient, high quality literacy programs; and - Increase in performance and satisfaction level of beneficiaries. 1Barnamaj Al Aoulaouaiyat Al Ijtimaiya or Social Priorities Program 1 7. These PDO indicators were informally redefined during implementation to allow for measurement. In the ISRs, the teams monitored: (i) the adult illiteracy rate; and (ii) the number of beneficiaries per year in Literacy Department (DLCA) sponsored programs. 8. Because of the pilot nature of the project, the PAD also cites a number of other indicators to monitor. The project aimed to demonstrate to what extent the measures put in place to revise the curricula and strengthen institutional capacity improve the overall quality of the adult literacy system. Project activities were to be assessed would regularly, and assessment criteria included: (i) the level of stakeholder satisfaction; (ii) learner enrollment rates; and (iii) dropout rates and success rates at the beginning and end of the project. 9. Firstly, "knowledge" from this LIL was to be measured in terms of: (i) the number and quality of curricula revised and training sessions given; and (ii) the range of measures put in place to improve the institutional capacity of the units in charge of the various programs and projects. 10. Secondly, the project aimed at contributing to decisions regarding subsequent scale-up, and to do so, the loan was to finance evaluations and determine whether: (i) users find that curricula have been adapted to their needs; (ii) teacher training and support have been institutionalized; (iii) literacy institutions and providers show increased capacity to manage program growth; (iv) an effective sector evaluation and monitoring system has been established; (v) unit costs are considered reasonable; and (vi) political will exists to support scale-up. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 11. There was no formal revision of PDOs and key indicators. 1.4 Main Beneficiaries 12. The primary target group was composed of the learners of the NGO Program for two reasons: (i) they were deemed to represent the majority of the learners in the medium term; and (ii) NGOs tend to target the poorest population, particularly rural women and girls. 13. In addition, NGO trainers, DLCA staff at the central and local levels, and to a lesser extent staff of public providers (MOE, Ministry of Agriculture, Entraide Nationale, etc.) would also benefit directly from training financed under the project. More generally, the whole community providing literacy services would benefit from improved curricula, and improved Monitoring and Evaluation (M&E). 1.5 Original Components 14. Component 1: Improving the Program Quality (US$ 3.79M). The objective of this component was to help improve the quality of literacy programs as the first stage in lifelong learning, so that they will have a lasting positive impact on the learners' social and economic lives. 1.1 Renovating curricula and teaching materials (US$ 1.8M). The objective of this subcomponent was to produce literacy curricula that meet learners' needs, and to supply teaching tools for the teacher training staff. These curricula were to be used in literacy projects operated by NGOs and other types of service providers. Activities included: (i) analysis of literacy curricula needs; (ii) revision and printing of existing literacy curricula; (iii) design of new specialized literacy curricula; and (iv) document collection. 2 1.2 Strengthening the teaching capacity of service providers (US$ 1.98M). The objective of this subcomponent was to put in place a teacher training system based on increasing the number of training sessions offered, so as to reach the greatest possible number of learners. These training sessions were to focus on those responsible for training teachers in the four types of programs. The activities of this component were: (i) Training of literacy trainers; (ii.) Training of senior trainers for educational reform; and (iii) Professional research and communication 15. Component 2: Strengthen Management of the system (US$ 0.76M). The objective of this component was to provide all levels of the system - in the central and deconcentrated offices of the MEFP of Labor and within the service providers themselves - with the capacity to manage large-scale programs so that Government's quantitative and qualitative objectives were attained. 2.1. Strengthening administrative management capacity. (US$ 0.19 M). This subcomponent was to strengthen the administrative management capacity of the various players so as to improve the effectiveness and efficiency of the partnership-based approach in developing literacy programs, particularly to improving the M&E systems needed for decision making. The activities of this component were: (i) development of an implementation plan for management of the deconcentration efforts; (ii) training for literacy project management; and (iii) monitoring and evaluation. 2.2 Strengthening local operations (US$ 0.57M). The objective of this subcomponent was to promote the emergence of local innovations, particularly within the NGOs, some one hundred of which are already involved in literacy efforts throughout the country. The following activities aim to strengthen these capacities: (i) literacy assessment training; (ii) post-literacy strategies; (iii) financing innovations. 16. Regular Evaluation and Analysis of Literacy Programs (US$0.77 M). In addition, the project included a third activity which involved a regular evaluation of project outcomes and an analysis of various aspects of the literacy programs, with a view to preparing to take successful innovations to scale. 1.6 Revised Components 17. There was no revision in project components. Specific project activities, however, were revised during the mid-term review. Under Component 1, the revision of existing literacy curricula was cancelled as the work was done at no cost by the DLCA. Under Component 2, the functioning of the innovation fund was redefined, and under Activity 3 under Component 2, the planned studies were cancelled. Other smaller adjustments were made. 1.7 Other significant changes 18. Scope and Scale. Due to modest project achievements in 2003 and 2004, slow disbursement, and some cost savings, the Mid-Term Review (MTR) carried out in September 2005 agreed on the need to: (i) cancel Euros 1.5 M (US$1.80 M equivalent) of the loan; and (ii) extend the closing date by 18 months, from June 30, 2006 to December 31, 2007. The Loan Agreement was amended accordingly in December 2005. 19. As a result of the partial loan cancellation, allocations to Components 1 and 3 were decreased, while allocation to Component 2 was increased, resulting in the following new amounts: - Component 1: Improving the Program Quality: US$1.48 M (from US$3.79 M) - Component 2: Strengthen Management of the System: US$1.79 M (from US$0.76 M) 3 - Component 3: Regular Evaluation and Analysis of Literacy Programs: US$0.19 M (from US$0.77 M) 20. Implementation Arrangements. The subsector went through important organizational changes that had an impact on project implementation. The project was to be executed by the Ministry of Labor, Professional Training, Social Development, and Solidarity (MEFP) through its Literacy Department (Direction de la lutte contre I'analphabétisme or DLCA). However, just after Board approval and before signing of the Loan Agreement, following the 2002 legislative elections and the constitution of a new Government, the DLCA was transferred from the MEFP to the MOE under a new Secretariat of State for Literacy and Non-formal Education or SEANF (November 2002), headed by a Secretary of State. 21. This change, which was in line with the recommendation of the 2002 IIEP Report (Evaluation des programmes d'alphabétisation des adultes au Maroc), had a substantial impact on project execution (see Section 2.2). The Loan Agreement, which was signed only in February 2003, after the promulgation of the 2003 Budget law, was slightly redrafted to reflect the establishment of the SEANF. Almost two years later, after the establishment of SEANF, a new Secretary of State was nominated following a Government reshuffle (June 8, 2004). 22. In October 2007, following the legislative elections, and two months before project closing, the SEANF was eliminated. The DLCA has since become a Department of the MOE. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 23. Lessons from Earlier Operations. The BAJ Basic Education Project (approved in May 1996 and closed in December 2003) included a literacy component (US$ 3.7 million) that was satisfactorily implemented (ICR, June 2004). The BAJ Basic Education project showed that NGO providers could reach at-risk populations. It also made evident two principal challenges that needed to be faced if the Government was to successfully meet the ambitious literacy targets set out in the National Education Charter. First, literacy achievement was disappointing as the approaches used were often based on instructional strategies of primary education. Second, the expansion of literacy programs meant a greater management burden at all levels of the system, and scarce resources needed to be used more effectively. 24. The LIL was designed to help the DLCA acquire the tools to meet both challenges by: (i) developing a variety of curricula geared to different populations, and (ii) reinforcing management capacity to improve program effectiveness. In addition, the project aimed at monitoring and evaluating these changes to assess whether the LIL-financed interventions would become the new management tools for the literacy sector. 25. Assessment of Project Design. The project addressed a Government priority, and responded to the DLCA's request to have a separate operation to support its activities. The project design was innovative and supported one of the key goals of the CAS Pillars, which was to encourage participation and inclusion. 26. The project design was consistent with the recommendations of a sector assessment done in 2002 by the International Institute for Education Planning (IIEP) (UNESCO) with Bank support and at the Government's request. The draft report findings were discussed during the February 2002 appraisal mission in a lively workshop gathering about 70 participants (including 4 representatives from various Ministries, NGOs, the private sector and international organizations). The report recommended inter alia2: Strengthening DLCA management and creating a sound deconcentrated network; Improving the quality of curriculum by responding to the specific pedagogical needs of diverse populations; Better monitoring of NGO activities and creating an incentive system for the NGO-supported program activities; and Improving M&E and research. The project supported the implementation of these recommendations. 27. However, the choice of a LIL as a lending instrument was not consistent with the project design. A LIL was not appropriate to support capacity building and quality enhancement of an entire program. Furthermore, in the absence of baseline information on learners and their pre- project learning achievement, the LIL hypothesis3 could not be tested. The two key performance indicators selected (i.e., increased participation of the poorest adults in efficient, high quality literacy programs, and increased performance and satisfaction level of beneficiaries) were not well-aligned with the objective of testing new approaches to literacy. It is difficult for a LIL to test new approaches and at the same time achieve major results regarding increased participation or increased performance and satisfaction during the life of the project. In fact, the PDO indicators showed some improvement when no implementation had really started (see section 3.2). 28. Assessment of Risks. The PAD rightly recognized insufficient government budget resources allocated to the literacy program as a risk. Inadequate budget resources were raised continuously by Bank teams during supervision. The Fall 2002 elections were mentioned as a risk as they could change the education plans spelled out in the Education Charter, but this was deemed unlikely to happen. Because it was difficult for the team to foresee at that time that the elections would lead, instead, to an institutional change affecting implementation during the first 2 years, this was not identified as a risk. Yet, during the QEA, the team acknowledged that the transfer of the DCLA to the MOE was an important risk to the success of the project (Office Memo, dated May 20, 2003). The institutional risk linked to the expansion of the programs was not identified. 29. Moreover, the institutional risk was not sufficiently analyzed. At preparation stage, the Country Director "urged the project team to consider more ambitious objectives aiming to support the Government's whole adult literacy strategy on a programmatic approach" that would include "an assessment of institutional capacity [of the MEFP] , fiscal impact [...] in moving from 200,000 to 500,000 beneficiaries per year" (Office Memorandum, October 12, 2001). Before appraisal, one peer reviewer also raised the need to pay more attention to the institutional structure under which the project would operate, as the DLCA was expanding its literacy programs at the same time as it was decentralizing many of its functions. These institutional issues were not sufficiently analyzed during project design. They became an even bigger challenge after: (i) the creation of the Secretariat of State which imposed additional institutional 2The report also recommended: (a) establishing a Secretariat of State for Literacy, or a Literacy Agency; (b) allocating adequate recurrent budget to the DLCA; (c) reconsidering the language of instruction; and (d) expanding and improving the private sector involvement. 3 The LIL was testing whether the measures put in place to revise the curricula and strengthen institutional capacity would improve the overall quality of the adult literacy system. 5 complications at the central and local levels; and (ii) the decision of the new Government to raise the already ambitious target of 500,000 adults trained per year to one million. These changes, occurring before signing, did not lead the team to reassess the project design, and the implementation agency capacity. Also, while the DLCA had no M&E indicator nor an M&E plan, the project did not include activities to build the system, which would have been a prerequisite for monitoring the impact of the LIL. 30. The May 2003 QEA 6 rated the quality at entry marginally satisfactory (3), equivalent to Moderately Unsatisfactory under the new QAG rating. The project was rated unsatisfactory on technical and economic analyses, institutional aspects, and analysis of risks and sustainability. 2.2 Implementation 31. Loan Effectiveness (May 5, 2003) was declared about 8 months after Board approval. There were no effectiveness conditions but the CPPR recommended project launch readiness checklist conditions (preparation of a set of bidding documents, and finalization of the project implementation manual). Implementation had two phases: a first phase during 2003-2004 where project activities were very limited (drafting/ revising terms of reference and adjusting consultants' contracts (to include some operating costs), and a second phase during 2005-2007 where most activities were implemented, many of them during the last year. 32. At the time of the planned MTR in September/October 2004, no disbursement had been made beyond the first advance to the special account, and no project activity had started (3 contracts out of 12 had just been signed). The MTR was carried out a year later (September 2005) and at that time, only 12% had been disbursed. The MTR concluded on the need to postpone the closing date by 18 months and to cancel Euros 1.5 million (US$ 1.80 million equivalent) of the project activities due to cost savings and slow implementation. While the project was implemented over 5 years, most of the activities were completed in the last years (Annex 2). At closing, total disbursement reached Euros 1.2 million (US$ 1.76 million equivalent), i.e., 28% of the original amount, and 44% of the revised amount. Three series of factors may explain this poor performance. 33. Factors outside the control of Government or implementing agencies. The main factor under this category was the difficulty finding qualified national consultants specialized in literacy. As a result, several bids had to be launched more than once, and some did not yield results. This factor was compounded by the limited budget allocated to some studies. 34. Factors generally subject to Government control. Creating the SEANF was a good decision that aimed at: (i) giving much-needed visibility to the literacy program, (ii) strengthening the DLCA ability to coordinate the programs, (iii) stimulating new and innovative initiatives, and (iv) mobilizing resources. The transfer of the DLCA to the SEANF, however, considerably slowed down initial implementation of the project for two main reasons. Firstly, the Secretary of State, nominated in November 2002, had limited ownership of the project. Rather than improving quality and efficiency, her priorities were to: reach the one million target announced by the GoM, develop a new literacy strategy following a broad consultation process, establish a deconcentrated network at the provincial and regional levels, reorganize the NGO program to allow participation of smaller NGOs, coordinate with public providers, establish a literacy passport, and mobilize budgetary resources. The new Secretary of State, who took office mid- June 2004, had a much stronger interest in project activities, and was able to push them forward. 35. Secondly, the SEANF initially had no separate budget from the MoE and no authority to commit its budget. All operational expenses had to be approved by the general accounting 6 division of the MOE which, given MOE's capacity at the time (which led the BAJ Project team to create a Project Management Unit (PIU) with staff attached to the civil service), and the difficult relationship between the Secretary of State and the Minister of Education, prevented the implementation of project activities. This situation was resolved in February 2004 when the SEANF received the authority to commit its budget. 36. Moreover, although a budgetary line for recurrent costs had been established specifically for the project, no funds were allocated until early 2005. The issue of counterpart funds was considered resolved by the Bank team towards the end of 2005. Factors generally subject to the implementing agency control. 37. Deconcentration of Functions without Adequate Capacity at Local Level. Just after the transition of the DLCA to the SEANF, the literacy agenda was moved away from the antennes of the MEFP and given to the délégués provinciaux of the MOE. Staff selected did not have an incentive to take on this additional burden and often did not have the capacity to do so. Around April 2004, the SEANF appointed 130 cadres (chefs de service) to serve as literacy and non formal education officers at regional (AREFs) and provincial (Delegations) levels. However, in the absence of an organizational chart for the SEANF, their responsibilities were not well defined, especially with regard to M&E. The chart, designed with support of the Bank team, was finally approved in 2007 and institutionalized the activities the cadres had been carrying out. These activities included technical supervision, administration, financial control and M&E, and were too large for one person, sometimes without secretarial support. This staffing constraint remains to be addressed. 38. Limited Capacity at the DLCA Central Level. Project implementation was affected by staff turn-over for key positions, lack of motivation and insecurity due to the institutional change and lack of incentives, as well as difficulty in recruiting high caliber candidates which led to insufficient capacity for administration and procurement. This resulted in delays in the preparation of procurement plans, and in hiring consultants, and repeated rejections of invoices submitted for payment to the Treasury (MOF) because of lack of conformity with World Bank procurement procedures. The Bank team tried to facilitate communication between the DLCA and the MOF, but the DLCA proactivity in implementing recommendations was lacking. At the technical level, the DLCA also faced difficulties in dealing with the complexity of the work required by the project. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 39. Design. The December 2004 mission Aide-Memoire stated that "the [DLCA] capacity to evaluate the impact of training activities and their cost is quasi-non existing [...] A strategy to define precisely the respective roles and responsibilities of the central and local level is still missing." Basic conditions to generate the information on the learning process required by the LIL were therefore absent. Focusing only on developing M&E tools and providing training, without attention to the general strategic and institutional framework, was insufficient. The project design should have included activities to build up the M&E system. 40. Implementation. Immediately after Effectiveness, weak M&E capacity was identified as a risk. In 2004, the Bank mobilized a Consultant Trust Fund to provide support to the DLCA on M&E to compensate for this design flaw. The Bank consultant was able to make a diagnosis of the M&E situation, and helped develop an M&E Plan, including the definition of key indicators, and a training strategy, while the development of software was supported by other donors. Some tools were produced (4 booklets and one CDROM), and were accompanied by training. The 7 project also financed key equipment (computer and printers) to support the M&E process within DLCA. The Bank continuously raised the issue of staff shortages at the local level, but without much impact. The annual Beneficiary Assessments (BA) was not carried out. A two volume BA was eventually produced at the closing date of December 2007. 41. Utilization. The tools for M&E were provided to the regional and provincial DLCA staff. However, their distribution to providers has been difficult because of lack of resources. Measures to assess the quality and the efficiency of the services delivered continue to need more attention. Finally, many of the indicators mentioned in the PAD were not monitored and the learning experiences could not be captured to lay the groundwork for a full-fledged program. 2.4 Safeguard and Fiduciary Compliance 42. The DLCA faced difficulties complying with the Bank procurement procedures, which was compounded by difficult communication with the Ministry of Finance. In 2005, the Bank assessed that DLCA needed one year on average to contract consultants. But given slow implementation, and the limited number of contracts to review, Bank missions often did not include a procurement specialist (only 2 missions out of 11 had one), and only one ex-post review was conducted. Most missions, however, included a procurement proficient Operations Officer who was able to advise the DLCA. During 2005-2006, a Procurement Specialist was based in the Country Office, and was able to provided more continuous support. 43. The DLCA also faced some difficulties on financial management. Un-audited interim financial statements were not systematically submitted as specified in the Loan Agreement (e.g., 2006). Annual audits, however, were carried out regularly by the Inspection Générale des Finances (IGF). 2.5 Post-completion Operation/Next Phase 44. There are discussions under the Morocco Basic Education Reform Program II- PARSEM II (P095714) Project preparation as to whether the Bank should continue its involvement in the literacy program. Although the continued importance of the issue of literacy in Morocco's human development agenda is recognized, the added value of the integration of a literacy program under this new operation is questioned. Given the past Bank interventions (BAJ Education Project and the LIL Project), the country team felt that the Government has already been able to develop a sounder approach to literacy with improved delivery mechanisms. The PARSEM II may be used to strengthen the capacity for M&E of the literacy program. 45. The EU is currently preparing a 17 million Grant that will likely pursue some of the LIL activities such as capacity building of central and local staff, and of service providers, M&E, etc. A MTEF for the literacy sub-sector has been prepared and indicates the need for additional resources to maintain an even more modest than expected increase in the number of beneficiaries over the next 4 years. Depending on the EU Grant specific activities, there may be a role for the Bank to continue its support to M&E which is an unfinished agenda and a requirement to improve literacy policies. Another area where the Bank could share international experience with the Government is the establishment of the planned Literacy Agency. Finally, the Bank could also: (i) mobilize around promising innovations (e.g., the ALEF pilot project4), or delivery modes (e.g., 4 E.g.: In the USAID/ALEF pilot project, teaching in Darija and Amazighe has helped reduce learning time (100 h to 60h) and increase retention (90%). 8 the introduction of ICT); and (ii) contribute to the review of the 2004 strategy for literacy and non formal education. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 46. Reducing literacy rates remains a priority for the Government. Improving quality of programs by adapting them to the specific needs of the population is an activity that DLCA continues to pursue under the GOM budget. The need to strengthen the management system based on deconcentration and partnership with various providers, and reinforcing the M&E system are still relevant, and may be reinforced by the EU program. Efforts to multiply and disseminate the tools produced by the project to the local level are ongoing. 47. Project objective, design and implementation are also valid under the current Country Assistance Strategy (Report No. 31 879-MA, June 2005) which supports: (i) the improved access to basic services, social programs, and economic opportunities in pockets of poverty and vulnerability; and (ii) improved education system efficiency. 3.2 Achievement of Project Development Objectives 48. The project was not able to test new approaches to literacy. Some tools were produced too late during implementation to allow for measurement, and for those that could be evaluated the M&E system was not adequate to measure their direct impact on program effectiveness. Yet, the two outcomes indicators monitored in the ISR showed substantial improvement. The number of people enrolled in literacy programs, and who are likely to be poor, was multiplied by 2.5 between 2002/03 and 2007/08, and the national illiteracy rate for adults aged 15 and above has been reduced from 50% to 43% between 2002/03 and 2005/06 (Table 1). The project has likely contributed to these outcomes, although they may have been influenced by other factors as well. In fact, the strongest increase in the number of people enrolled (57%) happened in 2003/2004 when project activities had not started. Table 1: Evolution of Project Outcome Indicators 2001/2002 2002/2003 2003/04 2004/05 2005/06 2006/07 Number of 390.000 286.425 450 000 470.000 655.478 709.000 beneficiaries per year in DLCA sponsored programs % increase 25.8% -5.62% 57% 4.2% 37% 8.2% Illiteracy rate (15+) 50% 43% Source : DLCA, 2008 9 49. The Project was able to produce most of the tools that may have contributed to these outcomes, and overall to the satisfaction of beneficiaries, as evidenced by the two volume Beneficiary Assessment 5 completed at Project Closing. The study's main conclusions are summarized in Annex 5. Specifically, the project supported the production of a good competency-based curriculum that has been used by the DLCA staff to revise (without consultant's support) the existing literacy curricula. These revised curricula are deemed to be relevant to trainers' needs. Yet they do not seem to be systematically used (60% of trainers use them, and 32% often use them). Training on use of the manuals was well appreciated by DLCA staff (91% satisfied), although coverage for providers was insufficient. Yet, the teaching guide raises some concern regarding its content, and its insufficient dissemination (67% of surveyed trainers not informed of its existence). Regarding the specialized literacy curricula (e.g., craft industry, tourism, and construction), DLCA staff, especially at the local level, feel they do not sufficiently take into account the sector needs. 50. Regarding the strengthening of the management system, the project was able to train 51% of total NGO literacy trainers in the country, with 95% satisfied or very satisfied, which represents a good performance. The project was also able to strengthen the capacity for deconcentrated management of DLCA staff and public providers with strong satisfaction although with regional disparities. More training on monitoring and evaluation, however, is requested by DLCA staff and public providers. Finally, although the project was able to train 200 managers as planned in literacy project management, only 32% of DLCA staff and only 15% of providers surveyed said they participated in training on administration and financial management of literacy projects. This underlines the need to scale up this activity in the future. 51. The report also indicates that 95% of primary beneficiaries felt that the literacy training was useful for their daily lives, and 72% that it helped improve their lives, with, some regional disparities. Finally, the project introduced a process of selecting innovative NGO sub-projects on a competitive basis. The innovation features mainly focus on increasing productivity and job opportunity. Because implementation is still ongoing, this activity could not yet be evaluated. The NGO classification method produced by the project now allows DLCA to select the more qualified NGOs, which should contribute to improve governance and to encourage NGOs to strengthen their professional skills for better teaching quality. 3.3 Efficiency 52. According to the IIEP study, in 1999/2000, the unit cost for the NGO-operated program was double that of the campagne générale, run by the MOE, and of other public provider programs (DH316 against DH150). However, there was no monitoring of the unit cost evolution during project implementation that could inform on efficiency gains. 53. Yet there is evidence that improved program quality and management resulted in efficiency gains. While the DLCA budget for literacy remained constant during 2003 and 2007, the number of people enrolled was multiplied by 2.5. Rough calculations indicate that the unit cost decreased substantially (see Annex 3). 5 Impact Evaluation of the Alpha Project, November and December 2007. A third volume is under preparation. 10 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 54. The project objective was generally relevant to the country's needs and priorities. Under the project, most of the tools to improve the quality of literacy program were produced, and overall, these tools meet the needs of beneficiaries, include their views, and are being used, with some degree of variation. Training was provided to 1,800 trainers, 200 public providers, 100 DLCA staff and 200 managers (DLCA and providers) to accompany the use of these tools and improve performance. Innovative NGO projects were initiated. 55. The direct impacts of these activities on teaching quality/learning and efficiency were not measured. The M&E system was deficient from the beginning and, despite improvements made during project implementation, it could not generate the necessary data to assess the achievement of the PDO. Yet, the reduction in the adult literacy rate, the increase in annual enrolment, and the efficiency gains under the NGO program likely result, at least partially, from project activities. 3.5 Overarching Themes, Other Outcomes and Impacts 56. Poverty Impacts, Gender Aspects, and Social Development. It is not possible to assess the project impact on gender and poverty. At the program level, however, women - who have a higher illiteracy rate (52% vs. 34.8% for men, 2006) were able to benefit predominantly: 82% of all literacy programs beneficiaries between 2002/2003 and 2006/2007 were women. In addition, half of the beneficiaries were living in rural areas where poverty and illiteracy, which are correlated, are more widespread (1998/99 LSMS). 57. Institutional Change/Strengthening. DLCA's delivery strategy is built on a partnership approach with other actors (NGO, MOE, other public providers, and the private sector). The institutional capacity built through the project has contributed to increasing the credibility of DLCA in its partnership with other providers, which may have led to the increase in delivery of literacy programs. More specifically, the project helped strengthen the country's capacity to implement overall higher quality literacy programs, except perhaps for the new specialized curricula (cf. BA). Although the project did not finance revisions of existing curricula, the Bank team provided technical support to do so, and the project provided a means to further improve it (competency-based curriculum). Yet more effort should be made to ensure a better use of these programs, and to disseminate the teaching guide which may be further improved at a later stage. The DLCA is now able to operate under a clear post-literacy strategy. M&E tools have been produced, but their multiplication and dissemination to trainers is an issue that needs to be solved. DLCA staff, public providers and 50% of all NGO trainers have been trained on M&E but at levels that remain insufficient. The project helped establish a functional documentation center that can be used by all providers and researchers. NGO small grants are now being completed under Government's budget, and may provide useful lessons. The DLCA will continue to implement this NGO innovation program in the future. The system established through the project to classify NGOs is now used under the NGO program. 58. Other Unintended Outcomes and Impacts (positive or negative). The Bank mobilized a CTF-financed international consultant to provide technical assistance on M&E activities but his role went beyond the project scope. His support was critical in developing indicators on access, quality and management of the literacy program; and in designing an M&E plan. The Bank also influenced several other areas, such as the initial strengthening of the DLCA local level, which may have accelerated the nomination of 130 cadres. 11 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops See Section 3.2 4. Assessment of Risk to Development Outcome Rating: Moderate 59. The risks to development outcome are deemed Moderate. Reducing illiteracy is an objective of the Education Charter that remains a valid strategy for the Government. Despite the absence of reference to literacy in the recent Emergency Plan to accelerate the implementation of the Charter, literacy is likely to remain a high priority in Morocco. To address the challenge of this sub-sector, the new Government announced the creation of a National Agency for Literacy for which statutes are currently being defined. It is hoped that this agency will be able to attract additional funding, and qualified human resources to further improve the efficiency of programs, which will involve disseminating and making full use of the tools produced under the project. Yet, the role and modus operandi of the agency vis-à-vis the existing structures of the MOE at the central and local levels remain to be defined. As a result, there is a risk of losing part of the project output, i.e., the strengthened capacity of the DLCA staff at central and local levels. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 60. The project was prepared after the successful experience of the literacy component of the BAJ Basic Education Project implemented by the DLCA. Therefore, the team could not easily anticipate the capacity constraint that the DLCA faced under the Alpha Project, although this project had more complex activities. The project represented a quick response to the country's request for assistance in the context of the sector dialogue; it was focused, not overly ambitious, and innovative. Yet, the institutional challenges posed by the entire literacy program and raised by the Country Director and a peer reviewer during preparation were not taken into account. 61. The QEA (May 2003) assessed the quality at entry "Marginally Satisfactory"6. A key area of concern was the use of a LIL as a lending instrument. In the absence of baseline information on learners, their pre-project learning achievement, and the capacity strengths and weaknesses of providers, the panel deemed that the LIL hypotheses could not be tested. The ICR team concurs with this assessment. Moreover, as there were no M&E indicators, strategy, and system, the project should have helped to build them from the outset. 62. The economic analysis and financial sustainability sections of the PAD were deemed insufficient. The ICR team also agrees with this assessment. For example, the LIL was about improving efficiency and the reduction in the unit cost is mentioned in the PAD as an indicator of project success. But there was no discussion of existing data (a rough assessment existed in the IIEP report) or any plan to monitor this indicator, and data today is still lacking. The panel also 6Equivalent to a "Moderately Unsatisfactory" rating in the new QAG rating structure. 12 felt the capacity building activities were insufficiently designed. This aspect, however, was not a determining factor in terms of project outcomes. 63. The PAD lacked clarity in presenting the facts, contextual conditions and project arrangements as raised by the QEA. The documentation shows some inconsistency in the PDOs, and the indicators (p2 and Annex 1). The choice of the PDO indictors is also questionable. They had to be redefined during implementation to allow for measurement. (b) Quality of Supervision Rating: Moderately Unsatisfactory 64. The Bank underestimated the initial impact of the transfer of the DLCA to the MOE Secretariat of State that occurred between Board approval and signing of the Loan Agreement on: (i) budget management; (ii) deconcentration arrangements; and (iii) reduced political attention to the project objectives. The team should have reassessed the project design and implementation arrangements at that time. It should have reassessed the capacity of DLCA to carry out the project, not only in terms of procurement and financial management capacity, but also at the technical level. The Bank team should have suggested hiring a consultant (procurement/administration) to follow up on day-to-day project management. 65. The team lacked candor in assessing project achievements. The Bank kept a satisfactory rating during the first 2 years despite the lack of progress, and the absence of disbursement. After some improvements in 2005, the new Team Leader downgraded the PDO to marginally satisfactory. Earlier realism could have led to a reassessment of the DLCA capacity to implement the project and find appropriate solutions in a timely manner. 66. It is only in April 2005 that the relevance of the project was re-examined. A good analysis of the reasons for the implementation delay was made, and scenarios for the future of the project were discussed with the GOM which led to the Mid-Term Review decision to cancel funds, and to extend the closing date. This coincided with the arrival of a new TL based in the Country Office (CO), who mentioned to the GOM the possibility of cancelling the Loan if no progress was made, and who was then able to provide frequent quality advice during 2005-2006. Implementation progress also coincided with the arrival in the CO of a Procurement Officer. The Bank team suggested the establishment of a Project Steering Committee headed by the DLCA Director, and organized monthly meetings with this Committee, an arrangement that provided the necessary momentum to improve project performance. 67. The Project had four different TLs during lending and supervision but hand-over was done appropriately and there was continuity in the implementation staff. To address M&E, the team was able to mobilize a CTF financed international consultant who provided extensive support to DLCA (7 missions during December 2004 - April 2007). He was also able to provide TA on a larger number of issues (including for example drafting TORs for studies and developing a Technical Note for the small grants to NGOs). Given the needs, DLCA and the project would have benefited from earlier and more continuous TA of this type. 68. References to Financial Management (FM) and procurement are not systematic in the Aide Memoirs (AMs), and only 2 missions out of 11 included FM and procurement specialists. Yet support was systematically provided by the Operations Officer who participated in all missions. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 13 69. The Bank's performance during preparation was adequate with regard to the relevance of the proposed activities but less so with regard to building the case for a LIL. During the first two years, supervision was strong technically but weak on implementation as project implementation arrangements were not reviewed after the change brought by the 2002 elections, and assessment of project performance lacked candor. The team at that time made some effort to raise the political ownership--although without success--and to facilitate the dialogue between the SEANF on the one side, and the MOE and MOF on the other side. Later on, the Bank addressed implementation issues by cancelling part of the loan, postponing the Closing Date and providing technical assistance either directly (during monthly meetings with DLCA in 2005-2006) or through frequent missions of the CTF funded consultant, in addition to regular supervision missions. These measures helped improve project performance from 2005 onwards, but to an insufficient level. As a result, most Project activities were completed late, while some remained incomplete. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 70. The 2002 Government was able to raise the visibility of the literacy agenda by: (i) establishing the SEANF, which may have been the first of its kind in the MENA Region; and (ii) announcing ambitious objectives (one million adults trained/year). 71. Yet, despite this new challenging goal, the budget allocated to SEANF remained stable over the project implementation period. DLCA had insufficient personnel and lacked logistical resources especially at the local level. In addition, the SEANF was unable until February 2004 to commit its budget which slowed down activities, and the first Secretary of State paid limited attention to the project activities. Counterpart funding was also lacking to finance transport and other stipends of training participants until end 2005. (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 72. The establishment of the SEANF mobilized DLCA staff at the central level. The Chef de division des programmes, appointed Project Coordinator, could not devote much time to the project. With the move of the DLCA under the MOE, the DLCA kept the same team but the accounting was managed by the MOE. High turnover and difficulties in recruiting skilled people resulted in weak administration, and in particular, procurement. 73. The dynamism created by the Secretary of State and the DLCA Director from 2005 onwards permitted the implementation of many of the project activities. Nevertheless, structural problems within DLCA limited the realization of the project's full potential after that date: lack of internal coordination between services, an organizational chart that did not facilitate the integration between literacy and post-literacy, and implementation plan and annual budgets not shared between actors to allow a regular administrative, financial, and technical follow-up. 14 (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory 74. The project interventions started at a difficult juncture. A new institution (SEANF) was being created and had to establish its operating environment (offices, budget, program, relationship with the MOE). Simultaneously, the same institution was given an unprecedented mandate for the country (training one million adults per year), and this without additional resources. These paramount challenges naturally relegated the project to a lower priority during the first two years. 75. Following this growth crisis, the arrival of a new senior management team at SEANF and DLCA, combined with the arrival of a CO-based TL, helped save the operation from cancellation. While the DLCA followed an accelerating path during the last 3 years, structural constraints and lack of skilled staff at the central and local levels of the administration have limited the possibility of regaining the time that had been lost. 6. Lessons Learned 76. In MICs, the incentive to invest in mastering Bank procedures is low for small capacity building loans. Morocco has a less than satisfactory performance in terms of disbursements for Bank investment projects, in particular when the Bank is disbursing against "soft" objectives such as capacity building. Investing, for example, in mastering Bank procedures to recruit consultants may be too costly for small loan amounts. 77. A LIL should be used to test new approaches, and not simultaneously to scale up project activities. Therefore, it is important to maintain as a priority the experimental aspect of a LIL and to not overburden it with expectations of going to scale. 78. Institutional stability is a condition for successful program scale-up. Major institutional changes, such as the establishment of the SEANF, can have an impact on the organization, human resources, financing mechanisms, and M&E system of a sector. Under these circumstances, scaling-up a program massively, while trying to improve quality and efficiency, is a daunting task. 79. Institutional change should be carefully managed to limit impact on program objectives. The transfer of DLCA from the MEFP to the MOE contributed to the lack of project implementation for the first two years. The Government will need to carefully manage the transition between DLCA and the Literacy Agency, technically and administratively, but also with regard to how this transfer is communicated to staff. More transparency on the preparation of the Literacy Agency would help reduce DLCA staff uncertainty about their jobs in the future. 80. Before a project begins, it is essential to undertake an appraisal of the capacities of the implementing agency. This lesson drawn from the BAJ Education project experience (cf. ICR, 2004) is also valid for this project. With the transfer of the DLCA to the MOE, the capacity to implement the project changed and should have been reassessed. This would have helped identify DLCA's needs for support in terms of project administration and procurement, but also on technical aspects, which could have led the Bank to recommend the help of one or two full time qualified staff or consultant to the DLCA, and to include fiduciary specialists more often in missions. 81. Candor should be used to assess PDO and Implementation Progress (IP) during project supervision. Assessing the PDO and IP with candor early on would have helped recognize the 15 project as a problem project, thereby motivating the team to provide a stronger and more effective response to the lack of progress on project implementation during the first two years. 82. It is not possible to test approaches under a LIL without an adequate M&E system. Building an effective M&E system is a pre-requisite, and should have been included in the project design. At project closing, the M&E system is still not operational. M&E tools have been produced but their use at the local level is not complete. As a result, basic data are not systemically collected, let alone analyzed. 83. When project outcomes largely depend on consulting services, it is necessary to analyze both the international and the national market for these services during preparation. Based on the findings, the Bank should agree with the Borrower on the share between national and international consultants. Also, consultant selection methods should be in line with the size and complexity of a project. Better use of the quality­cost based selection method should be made to allow for a stronger focus on quality. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies N.A. (b) Cofinanciers N.A. (c) Other partners and stakeholders N.A. 16 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Estimate Actual/Latest Components Percentage of (USD millions) Estimate (USD millions) Appraisal IMPROVING PROGRAM QUALITY 3.79 1.20 31% STRENGTHENING MANAGEMENT OF THE 0.76 1.60 210% SYSTEM EVALUATION AND ANALYSIS 0.77 0.30 39% Total Baseline Cost 5.32 3.10 58% Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 5.32 3.10 58% Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.04 0.00 .00 Total Financing Required 5.36 3.10 58% (b) Financing Appraisal Actual/Latest Source of Funds Type of Estimate Estimate Percentage of Cofinancing (USD (USD Appraisal millions) millions) Borrower N.A. 1.26 1.53 121% and Development International Bank for ReconstructionN.A. 4.10 1.57 38% 17 Annex 2. Outputs by Component Component/sub- Targeted Outputs Actual Outputs Comments component Component 1: Improving Program Quality 1.1. Renovating curricula and teaching materials (a) Analysis of A needs assessment; Competency-based Difficulty in literacy curricula development of a curriculum selecting a needs. competency-based completed in consulting firm; curriculum; evaluation of December 2007. first bid launched existing curricula, definition in 2003 (was the of TORs for activities 1.1.b. first project and 1.1.c. activity) but not successful; payment delays related to tax issue slowed down implementation. (b) Revision and 8 literacy curricula revised, 8 Completed in 2005. Given the delay printing of existing and a teaching guide These curricula have in doing activity literacy curricula produced been used since 2006. 1.1. (a), it was They are decided not to continuously invest in improved. consulting Teaching guide services and produced. revisions were done by DLCA staff. (c) Design of new Design 3 new manuals for Completed early specialized literacy the craft industry, tourism 2007 curricula and construction (d) Document Establish a document Completed in collection collection, and set up December 2007 collection management tools for DLCA, researchers, and service providers. 1.2 Strengthening the teaching capacity of service providers (a) Training of 2,400 trainers trained, esp. 1 800 trained. Request for literacy trainers those of NGOs Completed between proposals was mid-2005 and in June published twice; 18 2006 First time, bids were too high; still, Number of trainees was reduced to match the budget allocation to this activity. (b) Training of Train 20 senior staff from Completed in May senior trainers for DLCA at central and 2005 educational reform provincial levels through a 3 year continuous education program, including 5 workshops in Morocco; one workshop in Canada (b) (i) Training of 200 trained Activity added public providers Completed in during MTR (other ministries) on September 2007 the use of the revised curricula; (b) (ii) Training of 100 trained. Activity added DLCA Completed in 2007 during MTR. deconcentrated staff Done by DLCA on use of teaching staff at a reduced guide cost. (c) Professional Develop a strategy, and Not done research and create and publish quarterly communication newsletter Component 2: Strengthen Management of the system 2.1. Strengthening administrative management capacity. (a) Develop an A plan for the Plan was produced in Contract was implementation plan deconcentration of the early 2006, and signed in early for management of coordination, monitoring training was provided 2004. the deconcentration and evaluation of literacy to regional and effort. projects to regional and provincial staff of provincial entities is DLCA. produced and implemented. (b) Training for Needs assessment and Plan was produced in As for activity literacy project training plan on June 2004. 200 1.1. (a), payment management administrative and financial managers were delays related to management of literacy trained in 2006 tax issue slowed projects produced. 200 down managers (DLCA personnel, implementation. local offices of MEFP, service providers) trained on 19 administrative and financial management. (c) Monitoring and 200 managers (DLCA - tools to assess Activity was evaluation. personnel, local offices of quality of learning redefined in MEFP, service providers) developed + training March 2004 and trained in M&E, and of 100 central and during MTR. development of appropriate deconcentrated M&E tools. DLCA staff on the use of the tools completed in December 2007. - Guide for lit. project monitoring produced+ training of 100 staff Both completed in December 2007 - purchase of computers and printers for DLCA deconcentrated staff (2006, 2007) - GIS being developed 2.2 Strengthening Local Operations (a) Literacy 160 persons (DLCA Not done First bid not assessment training personnel, local offices of successful. Cost MEFP, service providers) deemed too high trained on survey methods for expected to assess literacy levels. benefits. (b) Post-literacy A seminar on post-literacy Seminar carried out strategies strategies (100 participants) end 2006 to review carried out,; national post-literacy 2 annual prizes (one experiences. Post ­ researcher, one NGO) literacy plan Prize not deemed established (with legal text). produced in 2007, a priority. and used. Prizes not created. (c) financing Finance innovations Small grants Activity was innovations designed at the local level to provided to 21 redefined during improve effectiveness of a NGOs. Sub-project MTR. Needs to literacy project. implementation still be evaluated (not ongoing. planned) Implementation manual revised; system of classification and qualification of NGOs designed; will be used in 2008. 20 Regular Evaluation 3 annual reports; 2 Reports were Studies were And Analysis Of Studies produced in canceled during Literacy Programs November 2007 and MTR. December 2007. Final report expected soon. 21 Annex 3. Economic and Financial Analysis It can be observed that from 2003 to 2007, while the national budget allocated to the DLCA remained constant, the total number of adult literacy learners was multiplied by 2.5 (Table 1). This comparison however may be misleading as the DLCA budget does not finance equally the four programs. The budget finances fully the NGO program and the campagne générale (run by the MOE), and partially other public provider programs (curricula, some training and in some cases trainers' fees) and the job-based program (curricula). However, the DLCA budget does not cover the program of the Ministry of Habous and Religious Affairs in which the number of people enrolled increased seven fold between 2003/04 and 2006/07 (from 19,960 to 137,970 people enrolled). Looking at the evolution of all other programs financed or partially financed by DLCA, one can still notice a 33% increase of the number of people enrolled under a constant budget. In 1999/2000 the unit cost for the NGO-operated program was estimated at DH316 by the IIEP study. A rough estimation of the evolution of this unit cost during the project implementation period shows a decrease from DH 351 to DH 180 per enrolled person during 2003 and 2007. This suggests that some important efficiency gains have resulted from improved program quality and management. Table 1: Evolution of the DLCA Budget and Evolution of number of participants in literacy programs 2003 2004 2005 2006 2007 % Increase 2003-2007 Budget (in million DH) 100. 6 88.2 92.9 103.0 103.0 % increase in Budget 8.2% -12.3% 5.3% 10.9 0% 2.3% Total Number of 286,425 450,335 469,206 655,478 709,155 people enrolled % increase -5.62% 57% 4.2% 37% 8.2% 147% Number of people [n.a] 430,373 447,659 522,202 571,183 33% enrolled, excluding the Religious Affairs Program Unit cost (in DH) 351 (*) 205 207 197 180 Source: DLCA, 2008 (*) based on the total number of people enrolled. 22 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending Jeffrey Waite Sr. Education Specialist MNSHD TL Dung-Kim Pham Operations Officer MNSHD Michel Welmond Sr. Education Specialist TL Monique Astalos Consultant Literacy Ferid Belhaj Lawyer Meryem Benchemsi Consultant MNCMA Financial Management Lead Financial Management Rafika Chaouli Specialist MNAFM Supervision/ICR Michel Welmond Sr. Education Specialist Siaka Bakayoko Sr. Financial Management Specialist MNAFM Meryem Benchemsi Consultant MNCMA Financial Management Fatiha Bouamoud Executive Assistant MNCMA Ousmane Diagana Country Manager AFMNE TL Amadou Wade Diagne Consultant AFCF1 Literacy/ M&E Alaleh Motamedi Sr. Procurement Specialist MNAPR Dung-Kim Pham Operations Officer MNSHD Maman Sidikou Consultant Literacy Anas Abou El Mikias Sr . Financial Management Specialist MNAFM Rachidi B. Radji Sr. Human Development Specialist MNSHD TL (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle No. of staff weeks USD Thousands (including travel and consultant costs) Lending FY02 11 61.92 FY03 3 17.78 FY04 0.00 FY05 0.00 FY06 0.00 FY07 0.00 FY08 0.00 23 Total: 14 79.70 Supervision/ICR FY02 0.00 FY03 13 49.89 FY04 11 75.90 FY05 18 65.71 FY06 17 77.67 FY07 15 78.47 FY08 3 14.34 Total: 77 361.98 24 Annex 5. Beneficiary Survey Results Key Conclusions of the Project Impact Evaluation Study ACTIVITY RESULTS Component 1: Improving program quality Competency based DLCA staff: satisfied (97%) but 1/5 feels it cannot be easily used by curriculum service providers, and trainers Providers: satisfied (95%) and large majority feel that it can be easily used. 94% feel it was used to revise the manuals. Revised existing DLCA staff. 85% feel that meet the needs and 68% feel that they are curriculum used by the trainers. Providers. 96% feel that revised manuals are always or very often used by trainers Trainers. 60% use them, and 32% often use them. 84% satisfied with content and form of revised manuals. Teaching guide DLCA staff. Mixed results on the content, 96% think it is used by trainers, 84% feel it meets the needs Providers. Mixed results on content. But used, and overall deemed useful to trainers (89%) Trainers. Only 33% of them informed about the guide. 93% of those informed feel it meets their needs. 33% always used it and 46% used is often. Specialized literacy DLCA staff: feel they don't take into account sufficiently the need of curricula (craft industry, the sectors. tourism, construction) Training on pedagogical DLCA staff: 91% satisfied. reform Providers: 95% NOT informed about the training. Among those who participated, 78% satisfied. Training of trainers NGOs Trainers: 51% of total NGO trainers in the country benefited . 95% satisfied or very satisfied. Yet only 41%-49% feel that the objectives of the training on andragogy, communication, and monitoring have been fully met (41%-47%: partially met) All trainers. Regarding training on M&E, 34% and 65% respectively feel it fully and partially met its objective. Component 2: Strengthen management of the system Strengthening capacity DLCA staff and public providers: 86% of satisfaction but large for deconcentrated regional disparities (less satisfaction in Fes Boulemane and Tadla management Azial regions, and oly 50% among DCLA staff). More training on monitoring and evaluation is requested. 25 Training for literacy Low participation: 32% of DLCA staff and only 15% of providers project management surveyed said they participated in training on administration and financial management of literacy projects. Were not informed. Moreover among those informed, 77%-72% not consulted to design training plan, and 64%-62% felt it was not adequate for their needs. Yet, the number of projects analyzed by provider per year was multiplied by 3 over 2002/03-2004/05. Note: It is interesting to note that at times, the DLCA and the providers were out of touch with the teachers/participants, as regarding the question related to the teaching guide: only 33% of teachers knew about them, and of these, only 33% used them; yet the DLCA and the providers were overwhelmingly convinced that the guides were used widely. There also appears to be low interest in the management training provided despite its importance to the management component of the project, but it is unclear whether this is because there was lack of commitment on the part of the expected participants or if this suggested perceived low quality and/or relevance of the proposed training. Source: Adapted from: DLCA, IIHEM Conseil, Evaluation de l'Impact du projet Alpha Maroc (Nov. 2007 and Dec. 2007). The study is based on a review of documentation (2002-2005), and on a qualitative survey carried out in 12 regions among 100 DLCA staff, 300 service providers, 700 trainers, and 3000 beneficiaries (has been carried out at different times since 2006). 26 Annex 6. Stakeholder Workshop Report and Results N.A. 27 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Except for three activities still under implementation ((i) design and development of a Geographic Information System for illiteracy, (ii) printing, and distribution of curricula and (iii) small grants for NGO innovative projects) all planned activities were completed. Project preparation, quality and content. The project benefited from excellent collaboration between the GOM and the Bank during preparation and appraisal. The project responded to GOM's priorities. Implementation was delayed due to: (i) a long preparation phase (2001-2003); (ii) no budget allocation for recurrent expenditure linked to project activities; (ii) complex and cumbersome IBRD procedures that were not mastered by DLCA; (iii) the need to use concomitantly national and IBRD procedures; (iv) multiples changes of the DLCA institutional framework; and (v) the qualitative nature of project activities which made implementation relatively difficult. PDO achievement. The project targeted the poorest zones and zones where illiteracy rates are highest; gender was also taken into account. The PDO is seen as being largely met given the expansion of literacy programs: the enrollment rate in literacy programs increased from 470,000 in 2004/2005 to 709,000 in 2007. This reflects (i) the Government's awareness raising efforts, (ii) a strategy based on partnership and the mobilization of all the actors of the four programs, as well as a focus on improving the quality of the programs. Concerning the project outcome, the BA indicates that the average dropout rate remained around 11% between 2003 and 2005, compared to 20% before 2003. Bank Performance. The performance of the Bank in assuring quality at entry is rated satisfactory. The Bank encouraged innovation throughout the life of the project and sought to draw lessons about alternative mechanisms for literacy programs that are developed in partnership between the private and public sectors and NGOs, and lessons about the best international practices in implementing (NGO sub-projects) and the integration of a competency-based approach for literacy programs. The quality of Bank supervision is rated satisfactory. Supervision missions were conducted regularly with adequately staffed teams. The mission findings brought forth issues that were critical to monitoring the project's performance. Borrower's performance is rated satisfactory. Overall DLCA implemented project activities in a satisfactory manner. Government participation in the project preparation is rated satisfactory. The Ministry of Finance adequately ensured the Government's contribution to the project, followed up and submitted the audit reports carried out by the General Inspectorate of Finance. Nevertheless, because certain actions or tools were only recently put in place, or recently completed due to delays in the execution of these activities, as mentioned earlier, it is likely that their effects on the quality of the programs will not be seen until at their full completion, i.e. in less than one trimester. 28 The loan disbursement level is considered moderately satisfactory. Difficulties became apparent and were, as indicated earlier, likely due to a number of factors related to the nature of the project, the institutional framework of DLCA and to the commitment and payment cycle of the Government Lessons learned. Planned activities were not implemented early enough to allow for an evaluation of their impact on the ground, but it is clear that the project contributed significantly to the creation of a positive dynamic by laying a strategic path for the sector that will promote the quality of literacy programs in Morocco. Unfortunately, inefficiencies were noticed in project administration, namely these relate to delays in implementing planned project activities. This is due to several factors: - The non-mastery of World Bank procedures and their incompatibility with national procedures; - Institutional instability in the sector at the start of the project; and - The innovative nature and the complexity of actions in the project framework (studies, NGO pilot projects, designing and setting up of the tools ...) made it difficult to identify resource people with the requisite competencies and leadership skills. 29 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N.A. 30 Annex 9. List of Supporting Documents World Bank World Bank ( 2002) Project Appraisal Document, Alpha Maroc Project, Report No. 23918- MOR World Bank (2004) Implementation Completion Report, Social Priorities Program (BAJ) Basic Education Project, Report No28397-MOR. Government International Institute For Education Planning, UNESCO (Sept. 2002) Evaluation des programmes d'alphabéstisation des Adultes au Maroc DLCA, IIHEM Conseil, Evaluation de l'Impact du projet Alpha Maroc (Nov. 2007). Rapport final de la Phase II, Evaluation continue du projet DLCA, IIHEM Conseil, Evaluation de l'Impact du projet Alpha Maroc (Dec. 2007). Rapport final de la Phase III, Evaluation continue du projet Royaume du Maroc, Conseil Supérieur de l'enseignement, Etat et perspectives du système d'Education et de Formation. Rapport annuel, 2008. Royaume du Maroc, Ministère de l'Education nationale, rapport d'achèvement et de résultats, March 2008. 31