Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) Report Number : ICRR0020327 1. Project Data Project ID Project Name P096998 CUSTOMS DEVT Country Practice Area(Lead) Kazakhstan Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-48740 31-Dec-2012 62,000,000.00 Bank Approval Date Closing Date (Actual) 27-Nov-2007 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 18,500,000.00 0.00 Revised Commitment 7,348,305.34 0.00 Actual 7,348,305.34 0.00 Sector(s) Central Government (Central Agencies)(100%) Theme(s) Tax policy and administration(33%):Trade facilitation and market access(33%):Other accountability/anti-corruption(17%):Legal institutions for a market economy(17%) Prepared by Reviewed by ICR Review Coordinator Group Jose M. Rodriguez Alvarez Clay Wescott Lourdes N. Pagaran IEGEC (Unit 1) 2. Project Objectives and Components a. Objectives According to the Loan Agreement, dated on February 2, 2008 (p.4), the project development objective (PDO) is: “to increase the efficiency, effectiveness and accountability of the CCC [Customs Control Committee] in order to: (i) promote internationally acceptable practices for expeditious processing of international trade flows, so as to further integrate the country into the world economy and improve the investment climate and competitiveness, (ii) improve taxpayer compliance with the Customs Code and thereby increase revenue collection, (iii) and support transparency of customs operations in order to reduce the potential for corruption and increase predictability for traders”. The PDO formulation in the PAD (p. ii and 5) is consistent with the Loan Agreement. This Review will assess achievement of the three parts Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) of the PDO, as stated in the Loan Agreement. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components 1) Institutional Development, Governance, and Human Resource Management, costing US$15.40m at appraisal, US$6.82m at completion, This component financed: (i) the comprehensive review of the CCC organizational structure including workload analysis and future staffing requirements as new systems and procedures are progressively implemented; (ii) a detailed review of CCC strategic planning methodology and delivery of training programs to assist the newly created Modernization and Strategic Development Unit; (iii) development of service standards and establishment of a modern management information system; (iv) review, redesign, development, and implementation of the National Integrity Development Plan within an effective anticorruption framework; (v) development and implementation of a human resource management strategy including the implementation of a merit-based human resource management system to facilitate a steady move towards a more professional Customs service, with appropriate skills, remuneration and other incentives and objective performance indicators, operating in a user-friendly manner; (vi) the development of a comprehensive training strategy and curriculum, based on a competency assessment and support for the establishment of the Customs Academy; (vii) the preparation of a comprehensive change management and continuous improvement strategy; and (viii) the implementation of a cost-based financial management framework that can be used to monitor the cost of customs services, as a requirement for World Trade Organization (WTO) accession. 2) Customs Operations, costing US$15.88m at appraisal, US$4.99m at completion. This component financed: (i) activities to align the code and secondary legislation with major international instruments and best practices; (ii) establishment of a special unit staffed with tariff, legal and systems-design specialists to build and maintain an integrated tariff system and development of e-document systems software; (iii) support to implement new measures of cargo control and declaration processing based on risk management including re- engineering and automation of the key business processes of cargo control, transit and declaration processing; (iv) strengthening of the back-end functions of post-clearance verification and audit through the design and implementation of a risk management system for post clearance verification and post-clearance audit and audit training; (v) strengthening of law enforcement functions through training and enhancement of support facilities; (vi) development of a comprehensive control and enforcement strategy to strengthen CCC capacity to manage compliance risks; and (vii) activities to strengthen interaction and collaboration within customs, with other border control agencies both within Kazakhstan and neighboring countries, and with regional and international institutions involved in international trade. In addition, this component financed a comprehensive review and rationalization of all import, export, and transit procedures and documentation across government agencies. 3) Information and Communication Technology (ICT), costing US$28.34m at appraisal, US$9.29m at completion. This component financed: (i) activities to strengthen CCC organizational capacity to plan and manage the ICT systems effectively; (ii) development and enhancement of an integrated ICT systems based on proven technologies used in modern customs administrations, and customized to Kazakhstan’s conditions, that will support an integrated tariff system, re-engineered customs processing, risk management, inter- agency data exchange, cross-border information exchange, transit control, and delivery of relevant training; and (iii) the strengthening of archiving, data warehousing and security capacity of CCC. 4) Program Management, costing US$1.21.m at appraisal, US$2.69.m at completion. This component financed operations and capacity building of the Project. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was approved on 11/27/2007, and became effective on 03/24/2009., Initial project implementation delays resulted in two project extensions. The first restructuring, approved on 02/06/2012 extended the project’s closing date for two years from December 31, 2012 to December 31, 2014. The second restructuring, approved 11/12/2014, extended the closing date by one year to December 31, 2015. The appraised cost was US$ 62m. The Bank provided US$18.5 million of the estimated total project cost at the appraisal, through a Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) Specific Investment Loan. US$43.5 million of financing was envisaged from the Government of Kazakhstan (GoK). Through an amendment of the Loan Agreement dated May 16, 2014, US$4m of the loan funds was cancelled. Total disbursement from the World Bank at the project closing were just 7.35 million, 39 percent of the revised loan amount. Actual counterpart financing was $16.45 million, 37 percent of the GoK expected contribution. The reduction in the costs are mainly due to the fact that, after the merger of the Tax and Customs Committees under the State Revenue Committee (SRC) in August 2014, and within the context of an economic crisis linked to the fall of oil prices and the depreciation of the local currency, the SRS decided to acquire the Automated System for Customs Data provided by the United Nations Conference on Trade and Development, and the development of the integrated customs ICT system included in the project was no longer considered relevant. 3. Relevance of Objectives & Design a. Relevance of Objectives The objective of the project was and remains relevant at closing. At the time of the project preparation (2006) economic growth had reached 10.7 percent, and the GoK viewed the integration into the global economy and regional partnerships (particularly the Eurasian Economic Community) as a prerequisite for a boost to economic development. GoK was negotiating the accession to the WTO, expecting as a result a significant increase of foreign trade. Within this context, the modernization of the CCC’s processes was considered necessary to facilitate trade as a key issue in the negotiations with WTO. The legislative framework was improved and strengthened by the approval of a new Customs Code in 2003. However, the reform agenda remained challenging and complex, and the GoK vision for the Customs Reform was included in the triennial plan Customs Modernization Program (CMP 2007-2010), and even emphasized in several speeches of the President in the Parliament. At time of the approval, the Customs Development Project (CDP) was aligned to the priorities of the Kazakhstan Country Partnership Strategy (CPS) 2005-2011, the second pillar of which addressed the promotion of competitiveness by strengthening the Government capacity to identify and reduce barriers to businesses and private investors, stating that “WTO accession and greater cooperation, particularly in areas such as customs and trade facilitation, are seen as vehicles to help Kazakhstan firms link into global value chains” (CPS 2005-2011, p. 5). These objectives remain relevant in the context of the current CPS for 2012-2017, which emphasizes as key priorities competitiveness and strengthened governance (p. i). Also the GoK Strategic Plan for Development 2020 outlines among its priorities advancing public sector reforms to increase efficiency, transparency and accountability, comprising business climate and improvements in the regulatory framework, building on steady progress made with public financial management reforms and in customs and tax administrations. This Plan is complemented with other major policy initiatives comprising enhancement of competitiveness. Rating High b. Relevance of Design The PDO was complex and ambitious. However, the project’s logical framework presents a clear and convincing casual chain, with relevant components, subcomponents and activities clearly linked to the PDO and its sub-objectives. This original design remained relevant for most of the project lifecycle. However, Component 3 (related to ICT) and parts of the Components 1 and 2 became less relevant as the institutional environment changed with the creation of the Eurasian Customs Union (ECU) in July 2010, and particularly after the merger of customs and tax administration in the State Revenue Committee (SRC) in August 2014. The Specific Investment Loan was appropriate for a project originally designed to provide a catalyst for introducing other major reform activities and further key investments in the development of modern material and technical infrastructure for Customs modernization. In summary, the design relevance is in the low side of substantial Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) Rating Substantial 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective Increased administrative efficiency Rationale The efficacy of the project in achieving this objective is measured by the following PDO indicators: reduction in processing time at border posts, reduction in clearance time at inland posts and increase in the percentage of revenue per staff member. In the case of reduction in processing time at border posts, there was a reduction in average customs processing time at border posts of 53%, short of the target of 75%, and with no evidence of attribution to the project. It is possible that interactions between the Bank and Government over the long period between project design and implementation could have influenced this and other achievements, but there is no evidence of this in the ICR. The design had hoped to achieving processing time reduction through an integrated ICT, e-declaration and e-audit system, but these were not fully implemented. In the case of reduction in clearance time at inland posts, it is also difficult to establish a causal link between its partial achievement and the project since the improvements were the result of the previous implementation of the new Customs Code in July 2010. In the case of increase in the percentage of revenue per staff member, there were significant achievements, but the baseline data cannot be compared with the results because revenues and staff numbers changed dramatically after 2014, when the customs and tax administration merged in the State Revenue Service (SRS). Along with this, the improvements in revenue collection are largely due to the establishment of ECU. In summary, the efficacy of this objective was modest because two of the three targets were only partially met, and achievements cannot be attributed to the project activities. Rating Modest PHREVISEDTBL Objective 2 Objective Increased administrative effectiveness Rationale . The CDP effectiveness was measured by two PDO indicators: physical inspections and reduction in the number of documents required for customs clearance. Although both indicators were achieved, these improvements took place before the start of CDP activities, and some key project’s components expected to lead to the achievements were not delivered. Physical inspections are taking place mostly due to the requirements of the ECU. The expected contribution from CDP was the improvement of the risk management system, but it was not completely implemented because the main ICT component of the project was Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) cancelled. Other third party sources also suggest that the actual level of physical inspections is much higher than the data reported by CCC first, and SRS later. In the case of the second indicator, the number of documents that the traders need to submit was drastically reduced from 14 to 3, in 2011, but this was before the project’s activities had been initiated. Beyond the indicators, CDP’s Components 1 and 2 contributed to institutional reforms that supported improvements in effectiveness, but were not captured in the results framework. Most of the recommendations related to institutional changes provided by these components were accepted by CCC, but only part of them were actually implemented, and they became less relevant after the merger of customs and tax administration in SRS, or because necessary regulation amendments never were adopted. CDP’s contributions to institutional reform include: 1) Training strategy and activities, in particular the establishment of a Regional Training Center designated by the World Customs Organization as an affiliate of the Regional Training Center for Central Asia, and an e-learning policy that includes an e-learning platform, manual and programs for the training activities. However, the Training Center did not collect performance data, a situation that makes impossible any assessment of the impact and quality of these e-training programs. 2) Organization reforms, comprising basically recommendations on CCC organizational design, anti-corruption and change management, HR management improvement, including training activities in planning and performance management. Despite some advancements, the establishment of SRS made less relevant part of them. 3) Business processes. There were advancements in business processes reengineering according to international standards, and training in risk management, the establishment of the integrated tariff system highly appreciated by the business community, post-clearance controls and audit, compliance management and the creation of an Authorized Economic Operator regime, among other achievements. However, the e-declaration was not fully rolled-out and never was implemented, and the new IC system planned for 2014 never was implemented during the project time-life. The ICR also mentions in the efficacy section the adoption of international standards and the facilitation of WTO accession, which can be mainly connected with effectiveness. Actually this was a more general objective of the GoK strategy for customs and trade facilitation policy, comprising other measures, among them the approval in 2010 of the Customs Code and its developments, with significant impact in Customs improvement. This new Customs Code, passed in July 2010, could not receive a serious input from CDP, since at that moment the project had just started its implementation). While accession to the WTO is considered part of this project’s intention to “promote internationally acceptable practices”, this achievement is the result of many efforts, including the general customs strategy of GoK and the support of diverse donors, and although there is a contribution of CDP to this achievement, taking into account the project’s results, it would not allow to modify the rating of this sub-component. In summary, although both targets linked to this sub-objective were achieved, there is no evidence of attribution to project support. Rating Modest PHREVISEDTBL Objective 3 Objective Increased administrative accountability Rationale Accountability is measured by two PDO Indicators: adoption of client service standards and perceptions of traders and stakeholders about unofficial payments. In the case of adoption of client service standards, CCC formalized six client service standards and published them in the SRC’s website, but even if they represent an advancement in the fields of transparency and accountability, these standards do not include a baseline or targets, a situation that makes it difficult to establish what should be considered good performance in this arena. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) With respect to the perceptions of traders and other stakeholders, there were two surveys funded by the project in 2011 and 2012, although ISRs reported data from 2013 and 2014. In the case of perceptions of traders and stakeholders about unofficial payments, there are not baseline data to measure the traders’ and stakeholders perceptions, and different criteria is used to measure their satisfaction. Alternative indicators, like those provided by the Business Environment Enterprise Perception Survey and the Association of Brokers, do not support positive trends in business perceptions. In summary, there is no clear evidence about the achievement of an increased administrative accountability and the rating of this sub- objective is also modest. Rating Modest 5. Efficiency The PAD (Annex 9, p. 76-79) included an economic and financial analysis with an expected Economic Benefit in terms of Internal Rate of Return (IRR) equal to 681%, and a financial benefit with an IRR equal to 170%. However, the ICR does not include an economic and financial Analysis. The project originally included equipment and infrastructure, and the economic benefits were expected from the reduction in the number of days required for customs clearance and the resulting reductions in costs for traders, to be achieved through a single window that never was implemented. As a result of that, the CDP did not achieve the expected economic benefits. With respect to the effects of the capacity building and the institutional reforms, it is very difficult to attribute them to the project and to measure the possible benefits obtained through them. The ICR noted, for example, that the reduced costs to traders due to the reduction in the amount of documents required for clearance cannot be attributed to the project itself. With respect to the possible gains linked to the use of distance learning in the Training Center -one of the project’s achievements-, the ICR does not provide any evidence about the quality of the learning activities. Finally, the delays in the project implementation: led to a significant increase in the planned management costs (component 4) by almost US$ 1.5 million, even though CDP only disbursed 51 percent of the original commitment. In summary, efficiency was modest due to the lack of achievement of the expected economic benefits linked to the establishment of an e- declaration, the lack of assessment on the quality of the e-training, and substantial additional management costs derived from project delays. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objective is high and that of design substantial. The objective was complex and ambitious, but well aligned with Bank and country strategies and relevant given the country conditions and circumstances. The casual chain between PDO, components and expected outcomes is substantial in the original design and remained relevant for most of the project’s lifecycle, although some activities became less relevant after the implementation of institutional changes related to Customs administration. Efficacy is rated modest in its three sub-objectives. The main shortcomings are the lack of achievement of a custom e-declaration, and lack of attribution of achievements to the operation. Efficiency is rated modest due to the negative economic impact of the lack of an e- declaration, the lack of evidence of an e-training quality, and additional costs linked to delays in the project’s implementation. a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating Taking into account that achievement of two of the PDO sub-objectives (efficiency and effectiveness) cannot be attributed to the project, while the risk of the third (improvements in accountability) is negligible, the risk to development outcome attributable to the project is also negligible. The future reforms will be informed by GoK plans to advance in the modernization of customs management, along with the modest CDP outputs. a. Risk to Development Outcome Rating Negligible 8. Assessment of Bank Performance a. Quality-at-Entry The Bank provided appropriate human and financial resources for the project preparation and appraisal (ICR, Annex 4) and CDP was aligned with the priorities of GoK and the CPS. The design of the project took into account the WTO capacity building framework, and drew on the previous experiences in customs reform, particularly in Europe and Central Asia region. However, although the objectives were outcome oriented, and the logical framework presented a relevant causal chain, most of the PDO indicators were not adequate for measuring the achievement of the objective, and the Bank team did not pay enough attention to the design of the project’s M&E arrangements. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Supervision missions were implemented 2-3 times a year, ISR and Aide-Memoires were prepared regularly, and a Mid-Term Review was carried out in December 2011. The bank team was complemented with Financial management and Procurement specialist in the field. However, the Bank supervision of M&E activities was much less effective, and the Bank did not propose a revision of the project indicators without taking into account the facts highlighted by the Mid-Tet Review on the basis of the available data. After the establishment of the ECU, the Bank did not consider restructuring the project to take into account the new institutional context and the requirement of the new legal framework. Also after the creation of the SRC, the Bank accepted the request of the GoK to leave both projects unchanged [CDP for customs Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) and the Tax Administration Reform Project for taxation] without any restructuring, when there was the need to align them better with the needs of the newly created SRC. In summary, during supervision the Bank team was neither proactive nor reactive enough to propose a needed restructuring, and the quality of supervision was moderately unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance During the project’s lifecycle GoK implemented significant reforms. Among other things, Kazakhstan accessed to the ECU (2010), created the SRS in 2014 merging tax and customs administration, and finally became a member of the WTO (2015). However, GoK failed in the implementation of the project, due to the lack of a strategic vision for the project. Senior officials contributed to the project’s design, but they did not pay enough attention to its adequate implementation. There was a significant delay (16 months) between approval and project effectiveness due to delays in the appointment of a project manager and in the ratification of the Loan Agreement. The creation of the SRS was not preceded by serious analysis, so the agency had to prepare the merger of tax and customs administration after its creation. Finally, GoK did not support the restructuring of the project suggested by the Bank after the creation of SRS, making a significant part of the planned procurement no longer needed. In summary, although GoK realized significant institutional reforms during the project’s lifetime, it did not adopt a proactive role during the project implementation nor did it react appropriately to adapt it to the institutional changes. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The Project Management Unit (PMU), once constituted, was generally dedicated, but a high leadership and staff turnover slowed project implementation and led to a loss of institutional memory, hampering timely decision-making. The delays were particularly significant in the bid evaluation and in the submission of bidding documents for Bank clearance. PMU staff also lacked management skills, and faced significant capacity constraints. The staff was more focused in the processes than in project’s objectives, but their lack of experience with Bank procedures along with the high turnover led to significant pitfalls, producing low quality important documents, like the terms of reference and other contract documents, which required external assistance from the US Agency for International Development to improve the quality of their drafting. The PMU also did not pay attention enough to relevant stakeholders like the business community and the public. According to the team, the rigid arrangements in the Loan Agreement about the distribution of expenditures between the loan and the country contribution, as well as the frustrated expectations of CMU staff about their salaries, also contributed to make the CMU management more complicated. Finally, the creation of the SRC, merging customs and tax administration, stalled the project implementation, and this fact, along with the significant reduction of funds allocated from the Budget as a result of the fiscal crisis, paralyzed the project implementation. Implementing Agency Performance Rating Unsatisfactory Overall Borrower Performance Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E design had significant shortcomings. As the ICR states (p. 90), “the M&E design of the project suffered from deficiencies which make it difficult to attribute the achievement of PDO indicator targets to CDP activities”. The PAD states (p.13) that PDO indicators were chosen “to provide a broader assessment of the achievement of key reform goals and objectives of the Government”, instead of measuring the project’s contribution to achieving the PDO. The project monitoring through the ISRs indicated that the customs reforms were making significant progress even before the initiation of the project’s activities. The Mid-Term Review noted that improvements followed the implementation of the new Customs Code in July 1, 2010, and the automation of the Customs Automated Information System, reducing significantly the clearance time for imports and exports, but these activities were not included in the project. This is an explicit recognition that achievements in PDO indicators 1 and 2 had nothing to do with the project implementation, but they were linked to other developments within the wider agenda on customs reform. In addition, the results framework did not specify the definition of the indicators included or in the methodology for collecting them, and as a result of that, there were significant differences between the data provided by different sources (CCC/SRS, stakeholders surveys contracted by the project, and other third party sources), since they used different methodologies. Finally, the PDO and the intermediate indicators did not measure important aspects of institutional strengthening and capacity building of CCC, first, and SRS, later, which were important aspects of the project. b. M&E Implementation The ICR emphasizes that neither the Bank nor the client paid enough attention to the M&E implementation. There are no comments on M&E in the ISRs, and in many cases comments on the indicators can only be found in the Aide-Memoires. There were also inconsistencies in the data provided by different documents (Aide Memoires, ISRs and Mid-term Review) and the data methodologies were not adequate to capture CDP/s changing context and circumstances. For example, in the PDO Indicator (Physical inspections), the data was no longer comparable after the accession of Kazakhstan to ECU, because after that moment clearance was not required for goods coming from Russia, the main exporter to Kazakhstan. c. M&E Utilization Although the PAD planned to build M&E capacities for the CCC (p.13), the training never was provided and CCC remained with limited understanding about M&E. The M&E reports planned in the PAD (p. 14) were not produced. As a result of that, neither the Bank nor the client (CCC/SRS) used the data on the PDO indicators to inform the decision making process. M&E Quality Rating Modest 11. Other Issues a. Safeguards According to the ICR, no safeguard policies were triggered by the operation. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) b. Fiduciary Compliance The rating of Financial Management and Procurement Management were always at least “moderately satisfactory”. There were no deviations or waivers from the Bank fiduciary policy requirements and the financial management arrangements provided adequate control and accountability through project accounting and reporting arrangements, internal control procedures and external audit. However, there were procurement capacity constraints, identified at the appraisal (such as the long internal approval process and the client’s limited experience with World Bank procurement guidelines), that were not properly addressed by CDP. The co-financing system with the country established in the Loan Agreement was very rigid, because every procurement procedure had to distribute the expenditures proportionally between the Bank loan and the Government contribution, which complicated the procurement management, particularly in a context of declining oil prices. c. Unintended impacts (Positive or Negative) None d. Other Some of the undelivered components from the CDP may be included in the on-going Kazakhstan Tax Administration Reform Project, according to indications from the GoK and the interview with the TTL. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Unsatisfactory Moderately Unsatisfactory --- Risk to Development Outcome Negligible Negligible --- Bank Performance Moderately Unsatisfactory Moderately Unsatisfactory --- IEG rates Borrower Performance as Unsatisfactory due to the negative impact in the Borrower Performance Moderately Unsatisfactory Unsatisfactory implementing agency management of the high staff turnover and the lack of management skills. Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The IEG concurs with the lessons drawn by the ICR (ICR p. 29), with some minor adaptations: • When the CMU manages significant ICT investments the role of an on-board, dedicated ICT Project Manager may be critical, in addition to the relevant ICT and procurement experts. In this case, the project design overestimated the CMU’s capacity to manage ICT procurement. • The revision of the monitoring and evaluation (M&E) arrangements during a project lifecycle is critical to diminish the chances that the PDO and/or intermediate indicators become irrelevant, and to add other pertinent ones, keeping a plausible connection between project Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CUSTOMS DEVT (P096998) components and activities on the one hand, and the M&E on the other. In the case of the CDP, the logical chain between PDO, components and activities was correct, but there was almost no direct connection between the PDO and the M&E arrangements made it almost impossible to prove the contribution of this project to the achievements in the indicators. •The assessment against various scenarios of the feasibility of mobilizing counterpart funds, when planned in projects financing, is a crucial element of the project design. In this case, the counterpart original planned financing was really significant, covering almost 70 percent of the project, but the macroeconomic risks were underestimated and a sudden change in the economic context (the dramatic decline of oil prices) led to a lack of counterpart financing of CDP. •When the project design ensures that different levels of government are involved in the policy-making process throughout its implementation, it is easier to deal with required institutional reforms and legislative requirements to support the reforms. In the case of CDP, many of the recommendations provided by consultants through the activities of Components 1 and 2 required formal legislation or high level Executive approval, but the rigid legislative environment in Kazakhstan and the lack of priority assigned by the Government to institutional development matters, CCC first, and SRC later, granted little room to achieve major successes in this field. A fifth lesson should be added: • Frequent changes and high turnover in Project Implementation Unit leadership and staff leads to slow project implementation and the loss of institutional memory, hampering timely decision making. This is what happened in CDP, with negative effects on project management. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR provided a clear, thorough, and candid assessment of the operation. The ICR provided an excellent example of the relevance of recognizing and analyzing a project’s shortcomings, particularly in relation to M&E design and implementation, the lack of links between the PDO indicators and the project results, and the analysis of the political and economic background and environment of the Bank operations and its evolution over the project’ lifecycle. From this perspective, the ICR positively contributed to significant lessons that should be taken into account in project design. As a minor issue, however, it is important to emphasize that the complete dates of some project’s milestones should have been provided and harmonized [for example, the restructuring key dates (p. iii), and p. 4-5 and in Annex 7 (p. 43)]. The ICR is in the high side of substantial. a. Quality of ICR Rating Substantial