Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) Report Number : ICRR0021294 1. Project Data Project ID Project Name P102624 IN:Kerala Local Govt. & Service Delivery Country Practice Area(Lead) India Social, Urban, Rural and Resilience Global Practice L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-48720 31-Dec-2015 260,000,000.00 Bank Approval Date Closing Date (Actual) 29-Mar-2011 29-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 0.00 Revised Commitment 192,775,692.08 0.00 Actual 169,987,287.97 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally Ridley Nelson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The project development objective (PDO) in the Project Appraisal Document (PAD, p.3) was identical to that in the Financing Agreement (p. 4). It was to: "enhance and strengthen the institutional capacity of the local government system in Kerala to deliver services and undertake basic administrative and governance functions more effectively and in a sustainable manner." Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The PDO was supported by four components: 1. Performance Grants (appraisal Cost: US$238.6 million, actual cost: US$235.88 million). This component would phase in an annual, performance-based grant to all 978 Gram Panchayat (GPs) and 60 Municipalities in Kerala. It would provide GPs and Municipalities with additional discretionary funds for expanded local investment in a manner that would incentivize the strengthening of their institutional capacity (supported, on the supply side, under Component 2). The performance grant (PG) would be spent on the creation, maintenance and operation of capital assets used in service delivery. The overall goal would be to improve GP and municipal performance in local governance and public service delivery. In order to provide a realistic and feasible mechanism for incrementally strengthening the institutional capacities of the GPs and Municipalities, the PG would be introduced in two phases. The administrative and institutional systems necessary for the introduction of a full performance grant would be established over the first two years of the project (Phase 1). During this time all GPs and Municipalities would receive grant funding, the use of which would be subject to basic fiduciary requirements. From year three onwards, the performance dimension of the grant, where qualification to receive a grant would be based on an annual performance assessment, would be applied (Phase 2). Component 1 would introduce the Performance Grant (PG) system as an integral part of the Kerala Local Self Government fiscal framework which would continue beyond the life of the project. New activities added under Component 1. In 2014-15 the project had accrued savings of US$62 million as a result of the depreciation of the Indian Rupee vis-à-vis the US dollar. These savings were utilized for: (i) a one-time additional financial assistance to 50 backward and tribal Gram Panchayats and 10 revenue- deficit municipalities for undertaking priority investment sub-projects focusing on service delivery improvement and local economic development; and (ii) Annual Performance Assessment 3 (APA) based Performance Grants to all local governments for an additional year. 2. Capacity Building (appraisal Cost: US$11.2 million, actual cost: US$8.10 million). This component would provide capacity building inputs to strengthen and supplement the existing systems and human resources of Local Self Governments (LSGs) to enhance their institutional performance. The focus would be in four core areas: (i) the development and improvement of administrative systems used by LSGs; (ii) formal training to strengthen the human resource capacities of LSGs using these systems; (iii) mentoring directed at human resource capacity improvement and supplementation; and (iv) strengthening of the overall institutional capacities of the key organizations responsible for delivering LSG training in Kerala. In the longer term, the component would support the formulation of a State-wide capacity building strategy for LSGs. Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) 3. Enhancing State Monitoring of the Local Government System (appraisal Cost: US$3.4 million, actual cost: US$1.00 million). This component would provide support to strengthen the system of performance monitoring of LSGs in Kerala. There were four sub-components: 3.1. Database of GP and municipal information. The project would establish a database of GP and municipal level information which would store basic information regarding GP and municipal profiles: population, vital statistics, livelihoods, employment, education, water and sanitation, budget expenditures and physical assets. 3.2. LSG Service Delivery Survey. A survey to gauge trends on LSG service delivery performance and citizen satisfaction throughout the state. The survey would examine delivery trends state-wide in sectors where GPs and Municipalities have some responsibility, explore citizen satisfaction and usage of basic services as well as awareness of GP planning and budget processes. 3.3. Project Evaluations. In addition, the project would plan two evaluations to evaluate the quality of the capacity building efforts (Component 2) and service delivery technical evaluation for the Performance Grant investments (Component 1). The latter would include an assessment of improved access to services as a result of the block grant investments; coverage/distribution of service provision; technical quality, operations and maintenance arrangements, cost effectiveness (value-for-money) and safeguards issues. 3.4. Decentralization Analysis Cell (DAC). The DAC would perform two key functions: (i) it would collect, store, compile and report GP/Municipality level and service delivery data; (ii) the DAC would also carry out a policy advisory function providing independent analysis on the performance of the State‘s intergovernmental fiscal system and service delivery system and provide ongoing policy advice to the Government of Kerala and the State Finance Commission on local and intergovernmental fiscal and institutional issues. 4. Project Management (appraisal Cost: US$6.80 million, actual cost: US$3.80 million). This component would provide support to the Project Management Unit (PMU) within the Local Self Government Department in overall coordination, implementation, monitoring and evaluation of the project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total project cost was estimated at US$260 million (PAD, p. 6). The actual project cost as reported in the ICR (Annex 1) was US$248.78 million. The difference was due to lower than expected actual costs for all four components; and cancellation of funds (see below for details). Financing. The project was to be financed through an IDA Credit worth US$ 200.00 million. The actual amount disbursed according to the ICR (p. 22) was US$182.83 million. The difference was because of the significant depreciation of the Indian Rupee (INR) against the US dollar from INR46 for US$1 (PAD estimate) to nearly INR65/67 per US$1 in 2017. As a result, the equivalent of US$7.2 million was cancelled from the Credit in May 2017. At project closure, the equivalent of US$6.3 million remained unutilized (which will be returned to the Bank), and an additional amount of US$ 6.5 million was incurred as an exchange loss by the Government of Kerala. Borrower Contribution. The borrower was expected to contribute US$60.00 million of counterpart funds. The actual amount contributed was US$78.80 million (ICR, p. 2). Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) Dates. The project was approved on March 29, 2011. It became effective on September 16, 2011; and was expected to close on December 31, 2015. The actual closing date was two years later on December 29, 2017. The ICR (p. 18, para 42) stated that "the extension of the Credit closing date was in part due to the original ambitious target of completing the entire project, including four cycles of annual performance grants, in a little over four years" and "the last two years of the project also witnessed two elections within a span of 18 months (each election imposed a three-month long code of conduct pause when, as per election laws, no new procurement could be initiated) and unusually long spells of monsoon rains (ranging between 3 and 6 months, when no physical works can be carried out on the field), which further added to implementation delays." The project was restructured twice, both Level 2 restructuring. The first was on May 24, 2017 when the amount disbursed was US$182.83, in order to extend the original Credit closing date (December 31, 2015) by 18 months till June 30, 2017; and to allow the Government of Kerala to complete new activities. There were also changes to the Results Framework and component costs. The second was shortly after the first on June 2, 2017, when the amount disbursed was still at US$182.83 million, in order to accommodate a second extension for the closing date by an additional six months till December 29, 2017 to allow local governments to complete the new activities in Component 1 (Note that the disbursement to them had already been made.) The Midterm Review (MTR) was expected to be conducted within 30 months of project effectiveness (PAD, p. 9, para 29). The ICR (p. 2) reported that the MTR was conducted on January 20, 2014, or 29 months into implementation. 3. Relevance of Objectives Rationale The state of Kerala has devolved more responsibilities and resources (funds, functions and functionaries) to local governments –and, within the local government system, to the lowest levels of local government – than any other Indian state. However, the state is facing three main challenges with respect to limited expenditure autonomy, weak institutional and organizational capacities, and inadequate state management and oversight mechanisms. Addressing these challenges is critical to empower local governments with the resources and capacity to deliver services in a sustainable and accountable manner within a strong and robust state level oversight and management framework. At project appraisal, the objective was highly relevant to the Government priorities at the state and national levels. In this respect, Kerala state’s Eleventh Five-Year Plan (2007-12) allocated 24.5% of its total plan outlay for decentralized planning and service delivery by local governments. The Objective was also in line with India's 1994, 73rd and 74th Constitutional Amendments, relating to decentralization and the empowerment of rural local government. The objective was also aligned with the third pillar of the Bank’s Country Assistance Strategy (CAS) for India for 2009-2012 which called for increasing the effectiveness of public service delivery, among other things. At project completion, the objective continues to be highly relevant to the Government priorities at the state and national levels. At the state level, the objective is in line with Kerala’s most recent five-year plan (2012- 17) and the ongoing fifth State Finance Commission (2016-21) that highlighted the importance attached to Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) deepening decentralization by empowering the local governments and strengthening their role, systems and capacities for efficient service delivery, both in terms of policy recommendations as well as higher fiscal transfers to local governments. At the national level, the objective continues to be in line with the Government of India's vision for development as outlined in India’s 12th Five-Year Plan (2013-2017) which called for faster, sustainable, and more inclusive growth, among other things. Also, the objective is in line with the most recent Country Partnership Strategy (CPS) for India for 2013-2017, which called for three main engagement areas: Integration, Transformation, and Inclusion. The CPS (p. 26) noted that "institutional development plays a key role in the case of spatial transformation, whereas strengthened accountability is critical for social inclusion;" and emphasized that the Bank would scale up support in performance-based public management, e-governance, as well as public and financial management. The objective is also in line with the challenges and priorities identified by the India Systematic Country Diagnostics (SCD) 2018 for the new Country Partnership Framework that is under preparation. The SCD (p. 33) emphasized the importance of empowering and resourcing local governments adequately for improving service delivery in a decentralized set-up. It also highlighted that incentivizing local governments in a competitive federalism framework can bring about a paradigm shift in their delivery of services (SCD, p. 80). Finally, the SCD identifies strengthening local governments as a key priority for strengthening India’s federal and public delivery systems. The stated objective was in line with country and state priorities, and included a reference to improved service delivery and governance functions - the outcome that was expected from institutionally strengthened local government systems (Gram Panchayats and Municipalitiess). Relevance of objectives is rated high. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective The PDO was to enhance and strengthen the institutional capacity of the local government system in Kerala to deliver services and undertake basic administrative and governance functions more effectively and in a sustainable manner. It included two elements: (a) - treated here below as Objective 1: Strengthened Institutional Capacity of the Local Government System in Kerala to Deliver Services More Effectively and in a Sustainable Manner. (b) - treated as Objective 2: Strengthened Institutional Capacity of the Local Government System in Kerala to undertake Basic Administrative and Governance Functions More Effectively and in a Sustainable Manner. Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) Rationale Outputs • Capacity building support was provided by Kerala Institute of Local Administration (KILA), State Institute of Rural Development (SIRD) and the Project Management Unit (PMU) through: the preparation of local government manuals, delivery of formal training sessions, and continuous mentoring/handholding support (provided by district coordinators at the local level) on service delivery systems (including detailed project report preparation, technical design and planning, environmental and social awareness, implementation and compliance with ESMF, procurement, contract management and asset management - ICR, p. 13, para 28). • 43,711 sub-projects (no target was set at appraisal) were built by GPs and Municipalities through block grants. These included financing 252 sub-projects for a select set of sixty backward, tribal and revenue- deficit local governments to enable them to implement additional local priority infrastructure projects to address their local economic and social needs. Sub-projects included: 1,593 Integrated Child Development Centers, 5,378 Buildings (Schools, local government offices), 709 hospitals, 541 bus stand/waiting shed/market, 139 burial grounds, 25,159 roads, 1,367 street light and solar energy, 1,350 drinking water supply, 7,475 other projects. According to the ICR (p. 18, para 46) at project completion less than 1% of the total sub-projects (approximately 323 sub-projects) remained incomplete. • 100% of GPs and Municipalities had their performance assessed each year. • 1028 LSGs received training over life of project in areas mentioned under component 2. • 12 draft manuals in the areas of budget, planning, financial management, public works/procurement, and institutional management (original target: 4). • GP database was established and operational with annual reports produced (target achieved). • 2 annual fiscal reports (target: 3) were provided by the Decentralization Analysis Cell (DAC). • 4 project studies/surveys were completed (target: 4). • Formal trainings delivered to all 1,028 local governments on key aspects of local governance and service delivery. • Systems and infrastructure of KILA and SIRD modernized and four Local government fiscal databases were prepared. • 27 policy research studies were completed, however, only 14 of these were published as the remaining 13 had concerns on their quality. • The number of direct project beneficiaries reached 29.5 million (target achieved) of which 52% were female beneficiaries. Outcome • Strengthen the institutional capacity of the local government system. The project contributed to local level planning and implementation systems through capacity building support and provision of performance grants. The Annual Performance Assessments (APAs) were used as a standardized mechanism to measure the performance of the GPs and municipalities in terms of their fiduciary, planning, governance, and service delivery systems in a fair and transparent manner. By project completion, the Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) number of GPs and municipalities that met all minimum mandatory conditions and scored above the threshold qualifying marks in the performance scoring was consistently in the range of 85-90% consistently exceeded the annual targets of 70% for PDO indicator 1. Also, 61 backward, tribal and revenue deficit LGs executed additional block grant support on time compared to a target of 51. • The impact of the local infrastructure and service delivery sub-projects implemented by local governments. Over 43,700 sub-projects were implemented by the local governments. These were identified through a bottom-up decentralized planning process with community engagement and people’s participation. Local infrastructure and service delivery sub-projects implemented by local governments were assessed in terms of real service delivery improvements on the ground, through an end-line technical service delivery evaluation was conducted by an independent agency. The survey examined 519 sub-projects, benefiting 3.6 million persons across 104 local governments, for technical quality, utility and usefulness of the public assets created, cost effectiveness, and compliance with environment and social safeguard norms. These sub-projects were found to be implemented within stipulated budgets and without cost overruns. The survey also found that monitoring systems put in place at the district level and the mentoring support strengthened the implementation capacities of local governments. The evaluation reported that construction quality was in compliance with technical standards/specifications and as indicated in the design document based on physical verifications of compliance with standards for: 82% of the road projects, 94% of public building assets and 95% of health care buildings. Finally, the evaluation reported that the public assets created through the assessed sub-projects had a high level of utility--where 98% of sample beneficiaries interviewed indicated that the public utility assets (mainly drinking water, roads, Integrated Child Development Service centers, public buildings and street lights) created under the project were being used regularly by their families. • The above mentioned information point to the success of the project in enhancing the institutional capacity of the local government system in Kerala to deliver services. However, sustainability needs more time to be accurately assessed. • Outcome is rated substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective (b) Strengthened Institutional Capacity of the Local Government System in Kerala to undertake Basic Administrative and Governance Functions More Effectively and in a Sustainable Manner. Rationale Outputs • Capacity building support was provided by Kerala Institute of Local Administration (KILA), State Institute Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) of Rural Development (SIRD) and the Project Management Unit (PMU) through: the preparation of local government manuals, delivery of formal training sessions, and continuous mentoring/handholding support (provided by district coordinators at the local level) on service delivery systems (including detailed project report preparation, technical design and planning, environmental and social awareness, implementation and compliance with ESMF, procurement, contract management and asset management - ICR, p. 13, para 28). • The project supported the development and integration of institutional systems for service delivery, financial management, financial reporting and monitoring project execution, through the Information Kerala Mission (IKM), including the creation of a local government fiscal database, the development of a comprehensive MIS system, and the integration of IKM software for local governments. • 1,028 LSGs received training over life of project in areas mentioned under component 2. • 12 draft manuals were produced in the areas of budget, planning, financial management, public works/procurement, and institutional management (original target: 4). Outcome The project succeeded in strengthening institutional capacity of local governments. This was achieved through extensive capacity building support provided to the local governments on key issues of local governance, local administration and service delivery. Annual financial statements and external audits were timely prepared by all the local governments as this was a mandatory condition to qualify for the project's performance grants. The ICR (p. 14, para 32) reported that the preparation of annual plans process was institutionalized and reflected high level of citizen engagement and community participation. Improvement in local government capacity was reflected in improved APA scores throughout the implementation period. Also, through project support the Procurement Manual was institutionalized by the state government, which all local governments were required to comply with for all procurement activities undertaken by them. According to the ICR (p. 15, para 33) this was expected to "improve local governance and service delivery through transparent and competitive procurement procedures." The project also contributed to streamlining and improving the local government functioning through the production of 12 administrative manuals. However, due to initial implementation delays, "training and handholding support to local governments for the adoption and implementation of these manuals at the local level was not adequate (ICR, p. 15, para 33)." An end-line capacity building evaluation conducted by an independent agency reported that survey of local government officials showed that 90% of them found that APAs had a positive impact on local government systems. The same surveyed group pointed out that APAs encouraged them to improve their institutional systems for local governance, administration and service delivery. These improvements enabled local governments to qualify for the project's performance grants. The evaluation also found that 75% of sampled participants across different categories found project trainings "useful, relevant and needed at that time....and formal training improved the overall efficiency of local government functioning (ICR, p. 15 para 34)." While the project supported a comprehensive MIS system, roll out and training of local governments was not completed due to implementation delays. Also, the Decentralization Analysis Cell (DAC) did not fully accomplish its envisaged role in providing policy advisory support to the state government and developing/maintaining a comprehensive local government fiscal and service delivery database. Institutional coordination issues played a major role in hindering the performance of DAC. Based on the information above mentioned information, outcome is rated substantial, despite delays in MIS Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) system roll-out. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale Both objectives were rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency ex ante • The PAD offered a very limited efficiency analysis. It did not include any quantified economic or financial analysis. It only argued that similar sub-projects such as those anticipated under the project had shown positive economic rates of return in other states in India. It is not clear what this meant since "positive" could either mean any positive number from a 1% ERR upwards or any positive number over and above the opportunity cost of capital, at that time generally taken to be 10% or 12%. • Efficiency gains associated with decentralization programs were considered difficult to quantify. Potential benefits included capacity enhancements to GPs and Municipalities and improved systems and processes. The assumption was that more efficient GPs and Municipalities would likely invest in local public goods that would carry a "positive" economic rate of return. The PAD argued that carrying out the project as designed through GPs and Municipalities was likely to give a better and more efficient impact than alternative selection and delivery modalities. This seems a reasonable assumption. ex post • The ex post economic analysis showed that the estimated Economic Internal Rates of Return (EIRRs) for the six selected case projects ranged between 15.5% and 43.5%. This is reported to be above the social Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) discount rate (SDR) of 8%. The ICR gives no references for, or evidence for the origin of, the 8% discount rate. It is not clear whether this has now been determined as an India-wide rate for all projects, or is project specific. • Assumptions: the analysis considered the stream of costs and benefits for selected cases over a 21-year period (2017-2037). Under the cost stream, construction cost during the one year construction period and annual / periodical maintenance cost during the remaining 20 years operation period from 2018 were considered. • The analysis focused on component 1 (performance grants) which utilized about 92% of the IDA Credit, to finance about 44,000 sub-projects. The six sample sub-projects, with costs ranging between US$2,000 and US$60,000, were selected, it appears, through a purposive selection process, covering six cases: construction of a new gravel road; converting an earthen road to a concrete road; rehabilitation of public wells to improve water supply; construction of a bus terminal; construction of a drainage system; and street light improvement. A purposive sample of six, one for each type, is an extremely small sample, moreover, one of each type did not reflect the predominance of roads at 60% of the expenditure, although there were two types of road construction cases included, so representing about 33% of the total by number of investment types. The ERRs are quite sensitive to changes in cost or benefit streams. For example, the predominant roads sub-projects, taking the concrete roads case, drops to 8.1% ERR, barely the assumed SDR, if projected benefits fall, say traffic volumes, by 20%. Given that this excludes project overheads, apportioning overheads such as management costs etc. would drop that activity below the SDR in the event that the benefit stream assumptions were too high in aggregate by 20%. • Specific benefits assumed included: construction of concrete / gravel road that would result in savings in vehicle operating cost; and savings in travel time cost for passengers; rehabilitation of wells for drinking water that would result in reduction in water collection time related benefit to the beneficiaries; construction of drain that would result in reduction in earning loss due to waterborne diseases caused by the waterlogging issues on the street; avoidance of flooding related issues during heavy rain and savings in earning loss during these flood days; construction of a bus terminal with link road that would result in savings in travel time cost for passengers; and replacement of CFL bulbs for fluorescent tubes that would result in savings in power consumption cost. • The ICR did not analyze in the sensitivity analysis the scenario of implementation delays and the impact of this on the rate of return. • The analysis did not include a quantification of the benefits of increased financial capacity and governance. Administrative and Institutional Efficiency The project closed two years later than expected. According to the ICR (p.18, para 42) this extension was needed to allow enough time to utilize the savings that accrued because of the significant depreciation of the Indian Rupee against the US dollar (the Indian Rupee depreciated from INR46 for US$1 (PAD estimate) to nearly INR65/67 per US$ in 2017). The equivalent of US$7.2 million was cancelled from the Credit in May 2017. At project closure, the equivalent of US$6.3 million remained unutilized, while an additional amount of US$ 6.5 million was incurred as an exchange loss by the Government of Kerala. Delays in the implementation Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) of Components 2 and 3 during the initial years resulted from complex implementation arrangements, with involvement of multiple institutions and conflicting reporting relationships, as well as weak inter-institutional coordination issues (ICR, p. 18, para 45). There were also significant delays at the state level procurement and contract management due to institutional coordination issues. These negatively impacted timely delivery of contracts which at times required multiple extensions (ICR, p. 27, para 83). On balance, Efficiency is rated substantial despite significant implementation delays, partly because some implementation delays appear to have involved both delays in initial expenditure as well as consequent delays in outputs, resulting in a lagged net benefit stream rather than a stretched out one, with different implications for an aggregate ERR. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 92.00 ICR Estimate  15.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated high. Overall efficacy is rated substantial, despite some shortcomings including the delayed roll-out and training for the MIS system. The project succeeded in developing institutionally strengthened local governments in the State of Kerala. This was accompanied by an improvement in the effectiveness of service delivery and basic governance and administrative functions. However, sustainability needs more time to be accurately assessed. Efficiency was rated substantial despite implementation delays and the lack of quantification of the benefits of increased financial capacity and governance. Based on the aforementioned ratings for efficacy, efficiency and relevance, outcome is rated satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) The sustainability of decentralization and strengthening of local governments is a long-term process. The Government of Kerala has demonstrated strong political will and policy commitment to decentralization efforts. These are critical factors to support on-going efforts, sustain on-the-ground results and further advance the decentralization agenda. Also, the Government of Kerala has shown interest in continuing the partnership with the Bank through a follow-on engagement to advance this agenda. The ICR (p. 29, para 94) correctly highlighted several actions that the Government needs to undertake to ensure sustainability. These include: • seeking to institutionalize the APA based performance grant allocation system by adopting it formally as a part of the state’s inter-governmental fiscal transfer system; • allowing local governments to earmark a portion of the development grants to operate and maintain local infrastructure assets finance under the project; • continue to provide capacity building support to local governments, • adopt and implement the 12 local government operations manuals prepared under the project, • complete the development and roll-out of the comprehensive MIS system; • and review and leverage the policy studies prepared by DAC to adopt policies that strengthen decentralization in Kerala. As in the case of West Bengal in India, the proposed follow-on Bank project would be a good vehicle to support the Government of Kerala in these tasks, especially for the institutionalization of the APA process in the fiscal transfer system of the state government. 8. Assessment of Bank Performance a. Quality-at-Entry • The project constituted one of four Bank-supported operations addressing decentralization and local governments in India(two were under implementation in Karnataka and West Bengal; a third was under preparation in Bihar). • Project objectives were in line with the country and state priorities. • Project design benefited from lessons and experiences of recent World Bank local government projects in India and in South Asia. The project drew on the experience of the Karnataka PRI project, which experienced a number of difficulties. • Notable lessons reflected in the design included: allowing complete discretion to local governments over performance grant expenditures; performance grants being disbursed in a single tranche at the beginning of the financial year and local governments being permitted to draw these funds according to their cash- flow requirements; and commitment of significant counter-part funding by the Government of Kerala. • Design featured a mutually reinforcing approach of capacity building support combined with performance based transfers. However, design was overly ambitious "in terms of the implementation period, weak PMU leadership at a critical period, and complex implementation arrangements (ICR, p. 18, para 46) and "inter- institutional coordination issues were not fully assessed and the reporting/accountability relationships were Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) not well defined (ICR, p21, para 57)." • At the appraisal stage 19 risks were identified and 17 of these had mitigation measures. The overall assigned risk rating was medium. The ICR did not explicitly report whether any of the identified risks materialized and if this happened, were mitigation measures adequate, a not uncommon omission in ICRs. • M&E design included a detailed Results Framework with defined baselines and annual targets, along with data sources and responsibilities for data collection/compilation and reporting. However, implementation suffered from inadequate mechanisms at the state and the local levels that negatively impacted periodic monitoring of project execution (see section 9 for more details). • The project was generally well prepared and design reflected a clear PDO supported by relevant components. Therefore, Quality at Entry is rated Satisfactory despite some shortcomings. Quality-at-Entry Rating Satisfactory b. Quality of supervision Over the six years of implementation, the Bank conducted fourteen implementation support missions that always included the project core members. These were supported by technical missions as needed to support and guide the PMU on key issues. Project implementation in the last two years benefitted from the presence of TTLs in Delhi. This ensured close support to implementation and timely responses to address any issues. Transition between Task Team Leaders was smooth through proper handover missions. The close supervision of the project ensured transparent and timely implementation of APAs as well as close consultation with high ranking government officials including the Chief Secretary and the Finance Secretary to resolve any issues (ICR, p. 28, para 90). The project benefitted from the Bank's Safeguards specialist who provided guidance and worked closely with the PMU on mainstreaming environmental, social and gender aspects in the project. Financial Management also benefitted from the Bank's support and close monitoring of fiduciary compliance of the project activities with Bank policies. Based on the above mentioned information, quality of Supervision is rated Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD did not include an explicit theory of change. Nonetheless the detailed project description (PAD, Annex 2) included clear and logical connections between the project inputs, outputs and expected outcomes. The Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) Results Framework (RF) included three PDO outcome indicators (including one core sector indicator). These were relevant, measurable and directly linked to the PDO. The RF also included 8 intermediate outcome indicators (two for each component) that would assess different activities under the project's four components. The RF clearly outlined units of measurement, frequency, data source, baselines (where applicable), and responsibility for data collection. The ICR (p.23, para 65) correctly pointed out that "annual targets were not defined accurately for one intermediate outcome indicator relating to the number and type of sub-projects being implemented by Local Governments, owing to the framework approach adopted for the project." A notable design shortcoming was the lack of a mechanism that would enable local governments to report on financial and physical progress of sub-projects. The Project Management Unit (PMU) had the overall responsibility of M&E. The Kerala Institute of Local Administration (KILA), State Institute for Rural Development (SIRD) and Gulati Institute of Finance and Taxation (GIFT) were each responsible for specific component-focused intermediate outcome indicators. Under Component 3 a, baseline and end-line service delivery surveys were expected to be conducted, as well as project evaluations to evaluate the results on service delivery and capacity building aspects. While Component 1 was expected to be monitored through a semi-annual administrative reporting system (ICR p. 23, para 65). b. M&E Implementation The PMU and the Decentralization Analysis Cell Project jointly carried out surveys and evaluations. While the end-line survey and the two evaluations were conducted and completed on time, the baseline service delivery survey was delayed due to initial implementation delays. The APAs were conducted annually and on timely basis. The implementation support agencies (mentioned above) provided information for intermediate outcome indicators for their respective components. Reporting on the achievements of sub-projects was undermined by the lack of a mechanism to monitor it by local governments; and the software used was not adequate to categorize different inputs. M&E quality suffered due to the vacancy of the M&E specialist position for a while during implementation. The situation improved towards the end of the project when a competent M&E specialist was hired. According to the ICR (p. 23, para 68) the PMU succeeded in collecting relevant information on all the sub-projects implemented under the project. While the project funded a comprehensive MIS system for local governments, its roll out faced significant delays due to coordination and software issues. The Information Kerala Mission "committed to complete the remaining stages of full system roll-out and handholding support to local governments under its own state funding (ICR, p. 24, para 69)." The Bank team and the PMU jointly developed M&E formats to closely monitor sub-project activities during the last two years of implementation. c. M&E Utilization According to the ICR (p. 24, para 71) "M&E reports were utilized by the PMU, execution support agencies, and the Bank team to make corrections and improvements in the work program." also, these reports were used by the Bank team to assess the project implementation and achievement of different output/outcome targets. The jointly developed M&E reports (between the Bank and the PMU) facilitated close monitoring of sub-projects as they were revised monthly. This enabled the detection of any issues that needed attention with regards to sub-project implementation. The project benefitted from the APAs as they provided key aspects of local government functioning. This information was critical for providing the Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) performance grants to eligible local governments; and for focusing capacity building interventions. M&E design was sound except for minor shortcomings and utilization was acceptable as the M&E data informed project management and was used in different sections of the ICR. However, implementation suffered from delays, lack of coordination and software issues which diminished the achievements of the design and it is thus rated Modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards • The project was in the environmental screening category B. It triggered the following safeguard policies: Environmental Assessment (OP 4.01), Forests (OP 4.36), Natural Habitats (OP 4.04), Physical Cultural Resources (OP 4.11) and Tribal Safeguard Policy (OP 4.10). • An Environmental Assessment (EA) was undertaken by the Government of Kerala (GoK). The EA identified the following key environmental issues concerning the activities of LGs: depletion of natural resources, changes in land use and topography, non-permissible activities in forests, natural habitats or areas of historical significance, soil erosion, inadequate waste disposal, usage of banned pesticides, and insufficient fodder resources. • An Environment and Social Management Framework (ESMF) was developed for the project (mainly for activities under component 1), to ensure compliance with the above-mentioned issues and triggered safeguards policies. No environmental impacts were envisaged under components 2, 3, and 4 of the Project. The ICR (p. 25, para 75) stated that "the project has complied with all triggered safeguards policies." • Environmental Safeguards. The ESMF was implemented successfully with all sub-projects screened for environmental impacts and appropriate mitigation measures applied (ICR, p. 25, para 75). While the Bank policy on Natural Habitats (OP/BP 4.04) was triggered, none of the sub-projects financed by the performance grants resulted in any adverse impact on any natural habitat. The Bank policy on Forests (OP/BP 4.36) was triggered and due measures were taken to prevent any conversion of forest areas or other adverse impacts. Bank policy on Physical Cultural Resources (OP/BP 4.11) was triggered and all relevant personnel involved in sub-project implementation were informed on the procedures to be followed to suitably protect monuments of cultural/historic importance; no adverse impact was found under any of the sub-projects. All sub-projects were screened and appropriate mitigation measures were incorporated as per the limited environmental and social assessments (LESAs) carried out, wherever applicable. • Two environmental audits were carried out during implementation, the first at mid-term in 2014 and the other at the end of the project in 2017. According to the ICR (p. 26, para 78): "the first environmental audit Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) found that ESMF compliance varied from 52.1% to 83.6% in most districts. However, the overall knowledge of Implementing Officers on the different aspects of environment was found to be less than satisfactory. A total of 99 sub-projects out of those audited were found initially to be non-compliant with some regulatory requirements. The audit found a number of design related shortcomings ... which tended to compromise the service delivery potential of the asset created or its sustainability." There were only three sub-projects with no compliance issues. However, these problems were later corrected in accordance with the ESMF. The second audit reported significant improvement in compliance due to capacity building activities that had led to better awareness among local government representatives. • Social Safeguards. The management of OP 4.10 Indigenous Peoples safeguard was part of the Vulnerable Group Development Framework. Annual reports captured how Scheduled Castes/Scheduled Tribes were being included in project activities. While OP 4.12 on Involuntary Resettlement was not triggered for the project; the PMU prepared an annual Land Database to inform the Bank on land use for sub-project activities. Land used for sub-projects was either already owned by the local government or donated, with proper documentation being available. The project benefitted from additional technical guidance provided by the Bank to improve knowledge and skills on the application of OP 4.12 in sub- project implementation. The Bank also worked with the PMU to address gaps in covering social safeguards aspects under the ESMF. Gaps in the ESMF and due diligence related to land were addressed in the revised ESMF. A number of sub-projects were delayed in the first round because of inadequate consultation with the communities and stakeholders involved. All local bodies had a system for receiving complaints and feedback from project beneficiaries; however, this system was not integrated in the project MIS. b. Fiduciary Compliance Financial Management. The PMU benefitted from robust staffing arrangements throughout the project period. Interim Financial Reports (IFRs) were submitted in a timely manner. While there were a few delays with the submission of the annual external audit reports, these did not exceed the Bank's deadlines. According to the ICR (p. 27, para 84), there were no serious issues raised in the external audit report. Internal audit arrangements experienced delays and weaknesses partly due to the Local Fund Audit (LFA) in audit certifications, and reconciliation issues. However, delays in internal audit process and delays in submission of internal audit reports by the State Performance Audit Office (SPAO) were systemic issues. Procurement. According to the ICR (p. 26, para 82) "compliance of the local governments with the procurement guidelines was high." The high level of community involvement in procurement, contract management and implementation supervision, contributed to an overall improvement in transparency and accountability. Social Audit Committees provided horizontal fiduciary assurance. Procurement operations benefitted from the decentralized procurement planning and implementation process. The Bank's annual post procurement reviews showed that the Local Fund Audit Team, "ensured vertical fiduciary assurance and also helped in concurrent monitoring that allowed local governments to make mid-course corrections (ICR, p. 27, para 82)." However, there were significant delays at the state level procurement and contract management due Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) to institutional coordination issues. This negatively impacted timely delivery and led to multiple extensions of contracts. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Bank Performance Satisfactory Satisfactory --- Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR included four lessons. The following are emphasized with some adaptation of language. • To ensure long term sustainability in local government and service delivery projects, capacity building of local governments should be an on-going activity delivered, where possible, through local institutes. In this case, the project supported the Kerala Institute of Local Administration and the State Institute for Rural Development (the key state-owned institutes) through infrastructure augmentation and systems modernization for delivery of training programs, curriculum development and strengthening HR capacities. This had a positive impact on their delivery capabilities and was favorably acknowledged by the state. This was one of the key sustainability measures put in place to ensure that the project-supported capacity building interventions (preparation of manuals, training needs assessment, training curriculum and calendar) and continued beyond the project on a sustainable basis. • To avoid conflicts and consequent delays during implementation, project implementation arrangements in local government service delivery projects should be simple and aligned with existing, or to be improved, government systems. In this case, the project implementation arrangements involved multiple agencies and the project reporting relationships conflicted with government structural reporting relationships. This led to institutional coordination issues and resulted in substantial delays. Also, implementation arrangements were not robust owing to weaknesses in the PMU, including frequent changes of Project Directors and many key positions remaining vacant for an extended period Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN:Kerala Local Govt. & Service Delivery (P102624) during project implementation. Implementation arrangements could also have been supported by stronger systems for monitoring, evaluation and reporting through a bottom up information flow approach. These posed challenges in monitoring the progress of sub-project implementation at the local level. Eventually the project had to rely on district coordinators offices, which presented its own constraints and challenges in terms of information flow. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well written. It provided concise coverage of project activities and candidly reported on most shortcomings. It also included a theory of change that clearly outlined the connection between project inputs, outputs, outcomes and expected impacts. Discussion of outcomes was logical and relied on the achievements of the project on the ground. The ICR also provided good coverage on Safeguards and Fiduciary. However, the ICR could have shed more light on the risks identified at appraisal and whether any of these materialized during implementation; and whether the suggested mitigation actions were effective. It could have attempted a larger sample than the very small six cases for the economic analysis, which, in any case, did not aggregate the ERR to include overhead project costs. It could also have clarified where the 8% Social Discount Rate came from, whether from a country, generic project-type, or project-specific, analysis, and what methodology was used to derive it. a. Quality of ICR Rating Substantial Page 18 of 18