The Document of The World Bank Report No: ICR 2976 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-41160) ON A CREDIT IN THE AMOUNT OF SDR 60.80 MILLION (US$ 93.72 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR AN INFORMATION AND COMMUNICATION TECHNOLOGY DEVELOPMENT PROJECT June 26, 2014 Transport, Water and ICT Department Vietnam Country Management Unit East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2013) Currency Unit = Dong (VND) VND 0.000047 = US$ 1.00 US$ 1.00 = VND 21,100.00 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CAS Country Assistance Strategy CA Certification Authority CCNA Cisco Certified Network Associate CERT Computer Emergency Response Team CIO Chief Information Officer DCA Development Credit Agreement DIC District Office of Information Communication DOT Department of Transportation EA Enterprise Architecture e-GAF e-Government Assessment Framework G2B Government to Business G2C Government to Consumer/Citizen G2G Government to Government GIS Global Information System GOV Government of Vietnam GSO General Statistics Office HCMC Ho Chi Minh City ICT Information and Communications Technology IDA International Development Association IO Information Officer IOI Intermediate Outcome Indicator LAN Local Area Network MAN Metropolitan Area Network MCSA Microsoft Certified Solutions Associate M&E Monitoring and Evaluation MIC Ministry of Information and Communication MOFA Ministry of Foreign Affairs MPI Ministry of Planning and Investment MPT Ministry of Posts and Telematics ODA Official Development Assistance PAPI Public Administration Performance Index PARI Public Administration Reform Index PCI Provincial Competitiveness Index PCU Project Coordination Unit PDO Project Development Objective PIU Project Implementation Unit PSO Provincial Statistics Office RF Results Framework SAMCOM State Administrative Management Computerization SHS Statistical Hub System SME Small and Medium-sized Enterprise SSIC System of Statistical Information Collection UNFPA United Nations Population Fund WAN Wide Area Network Vice President: Axel von Trotsenburg Country Director: Victoria Kwakwa Sector Manager: Randeep Sudan Project Team Leader: Deepak Bhatia ICR Team Leader: Deepak T. Bhatia VIETNAM ICT Development Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 5 3. Assessment of Outcomes .......................................................................................... 14 4. Assessment of Risk to Development Outcome......................................................... 22 5. Assessment of Bank and Recipient Performance ..................................................... 23 6. Lessons Learned ....................................................................................................... 27 7. Comments on Issues Raised by Recipient/Implementing Agencies/Partners .......... 29 Annex 1. Project Costs and Financing .......................................................................... 30 Annex 2. Outputs by Component ................................................................................. 31 Annex 3. Economic and Financial Analysis ................................................................. 63 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 68 Annex 5. Beneficiary Survey Results ........................................................................... 70 Annex 6. Stakeholder Workshop Report and Results................................................... 78 Annex 7. Summary of Recipient’s ICR and/or Comments on Draft ICR .................... 79 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 90 Annex 9. List of Supporting Documents ...................................................................... 91 A. Basic Information Vietnam - ICT Country: Vietnam Project Name: Development Project ID: P079344 L/C/TF Number(s): IDA-41160 ICR Date: 06/26/2014 ICR Type: Core ICR THE GOVERNMENT Lending Instrument: SIL Borrower: OF VIETNAM Original Total XDR 60.80M Disbursed Amount: XDR 48.01M Commitment: Revised Amount: XDR 52.04M Environmental Category: C Implementing Agencies: Ministry of Information and Communications Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 04/11/2003 Effectiveness: 09/29/2006 09/29/2006 01/28/2010 Appraisal: 05/23/2005 Restructuring(s): 06/29/2011 06/20/2013 Approval: 09/15/2005 Mid-term Review: 03/16/2009 03/16/2009 Closing: 06/30/2011 12/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Substantial Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Moderately Quality at Entry: Government: Unsatisfactory Unsatisfactory Moderately Implementing Moderately Quality of Supervision: Unsatisfactory Agency/Agencies: Unsatisfactory Overall Bank Moderately Overall Borrower Moderately Performance: Unsatisfactory Performance: Unsatisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Unsatisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 43 50 General industry and trade sector 22 10 Sub-national government administration 35 40 Theme Code (as % of total Bank financing) Administrative and civil service reform 25 55 Infrastructure services for private sector development 25 25 Micro, Small and Medium Enterprise support 24 10 Other accountability/anti-corruption 13 5 Regulation and competition policy 13 5 E. Bank Staff Positions At ICR At Approval Vice President: Axel van Trotsenburg Jemal-ud-din Kassum Country Director: Victoria Kwakwa Klaus Rohland Sector Manager: Randeep Sudan Pierre Guislain Project Team Leader: Deepak T. Bhatia Zhenwei Qiang ICR Team Leader: Deepak T. Bhatia ICR Primary Author: Anna F. Roumani F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The objectives of the proposed project (as stated in the PAD) were to: (a) accelerate the use of ICT and support institutional development in order to increase the efficiency and effectiveness of public administration, increase transparency and availability of information, and improve public services delivery to stakeholders; and (b) to foster private sector development in particular by facilitating the increased adoption of ICT by SMEs. Revised Project Development Objectives (as approved by original approving authority) A Level One Board-approved Restructuring in 2010 revised the PDO (and Key Performance Indicators) to focus more narrowly on project activities funded by IDA by assisting the Borrower to: (a) support MIC's institutional development and technical and regulatory capacity for ICT sector oversight and to increase the efficiency and transparency of public service delivery in MIC, GSO, MOFA, Hanoi and Danang; and (b) foster private sector development by increasing access to business-related government services and raising awareness and usage of ICT in Hanoi and Danang. Further adjustments to PDO and Intermediate Outcome indicators were made via a Restructuring in June 2013, approved by the Regional Vice-President. For clarity and consistency with the ICR Guidelines Appendix B requirements, the presentation below is organized as follows: (i) original, PAD-based indicators are listed first in both sections (a) and (b) and are used to measure the initial phase of the Project; (ii) the revised indicators resulting from the 2010 Level One Restructuring are then shown in follow-on sections of (a) and (b) immediately under the PAD originals; (iii) the June 2013 adjusted indicators and results are shown in the "Formally-revised Target Values" column of these two post-Restructuring sections and are used to measure achievement of PDO in the Project's second phase. (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Original Key Performance Indicators (from PAD): Indicator 1 : Increase share of ICT sector in GDP by 2 percent. Value Zero (PAD did not state 2% increased quantitative or existing ICT share of na share Qualitative) GDP) Date achieved 05/23/2005 06/30/2011 01/28/2010 Not achieved: Indicator was not included in PAD Arrangements for Results Comments Monitoring (ARM). Required sophisticated measurement not feasible within life (incl. % of project, difficult to attribute to project, and lacked baseline. Dropped by the achievement) 2010 Restructuring. Indicator 2 : Increase in ICT capacity and knowledge in respective government departments. Value quantitative or Zero Undefined na Qualitative) Date achieved 05/23/2005 06/30/2011 01/28/2010 Comments Not achieved: Indicator was not included in ARM, scope was too broad, target (incl. % not quantified and indicator dropped by 2010 Restructuring without achievement) measurement, for same reasons as above. Indicator 3 : 15% use ICT for business operations in 2010 and improve productivity Value 3% SMEs use e- 15% and improved quantitative or commerce (2003 survey na productivity Qualitative) data) Date achieved 05/23/2005 06/30/2011 01/28/2010 Comments Not achieved: Indicator was not included in the ARM, scope was too broad and (incl. % meaning unclear. Dropped by 2010 Restructuring without measurement. achievement) Indicator 4 : Domestic IT service sector grows by an average of 30% every year Value 30% growth quantitative or Zero na annually Qualitative) Date achieved 05/23/2005 06/30/2011 01/28/2010 Comments Not achieved: Indicator not mentioned in PAD ARM. Broad scope, project (incl. % attribution issues and lack of baseline constrained measurement. Indicator achievement) dropped by the 2010 Restructuring. Increase in standardization and improvement in transparency of administrative Indicator 5 : processes Workflows uncoordinated 100% of identified Value and fragmented due lack workflows and quantitative or na of integrated computer reporting lines Qualitative) systems automated Date achieved 05/23/2005 06/30/2011 01/28/2010 Comments Not achieved: Indicator too broad and attribution to project problematic. Re- (incl. % phrased by the 2010 Restructuring and combined into Intermediate Outcome achievement) Indicator (IOI). See IOI 40 below. Significantly lowered transaction costs for targeted government services (G2C Indicator 6 : and G2B) E-services were in process of selection Lengthy transaction times (radio spectrum, Value and high costs (formal publications, quantitative or and informal) associated Undefined standards for Qualitative) with many government integrated services communications equipment). Date achieved 05/23/2005 06/30/2011 01/28/2010 Negligible achievement: Services still being selected at time of Restructuring Comments (2010). Baseline and target not quantified. Indicator was revised by the 2010 (incl. % Restructuring and became an IOI linked to MIC modernization. See IOI 42 achievement) below. Indicator 7 : Increase in user satisfaction for e-government services and visits to City Portal. Value Low satisfaction among quantitative or users of many existing Undefined na Qualitative) government services Date achieved 05/23/2005 06/30/2011 01/28/2010 Comments Not achieved: Indicator was not mentioned in ARM, too broad, and baseline not (incl. % quantified to measure incremental gains. Revised by the 2010 Restructuring. achievement) See Results Indicator 9 below. Results Indicators (RI) Post-Restructuring: (01/28/2010 to 12/31/2013) Indicator 8 : Improved formulation and implementation of ICR policies and common standards Adjusted RI 2013: Formulation, 2010 Indicator: implementatio Priority telecom n ICT policies, policies and Limited capacity to common implementation 4 standards formulate policies and standards in Value guidelines formulated: EA, guide implementation on MIC. Target: quantitative or formulated; Inter-operability; E- latest issues in ICT sector. 3 standards, Qualitative) priority policies Government; and Gap in policy Enterprise for information Security Standard instruments. Architecture technology and IT (EA); Inter- industry promotion operability; in place. and, E-Govt Framework and Security. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Achieved 100%: The 2010 indicator was clarified in 2013 to improve link to the (incl. % PDO and measurement. Actual baseline at the time was: EA and Interoperability achievement) already formulated; e-Gov Framework and Security under implementation. Reduced transaction costs, increased transparency and users' satisfaction of Indicator 9 : selected government services through e-applications Adjusted RI: Reduced time Danang: 1 G2B for one service operational transaction of (Bus. Registration); selected govt devt phase services Lengthy transaction times completed, system Value through e- and high costs of many launched. No time quantitative or Undefined applications. government services, and for user survey. Qualitative) low user satisfaction Hanoi: 3rd level Target: public services Transaction online via Hanoi time reduced DIC; transaction by 1.5 times times reduced 60% (Hanoi, Danang) Date achieved 01/28/2010 06/30/2013 12/31/2012 12/31/2013 Comments Achieved 100%: 2010 indicator was clarified in 2013 and scope narrowed. (incl. % Specific elements (transparency, user satisfaction) became separate IOIs by city. achievement) Cost and user satisfaction surveys not completed under this RI. Indicator 10 : G2B services made available online (Hanoi and Danang) Adjusted RI 2013: No. G2B services available online (Hanoi, Danang). Danang: One G2B One G2B service Target: 3 G2B Value service (Bus. operational (Business services quantitative or Undefined Registration) Registration Service in online; Bus. Qualitative) launched and Danang) Reg. Service available online. (Hanoi, Danang); Public Procurement Service (Danang) Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Partially achieved: 33% of revised target. 2010 target clarified and quantified in (incl. % 2013. Public Proc. Service in Danang cancelled due implementation with Govt achievement) funds. Hanoi Business Registration procurement package HAN7 cancelled. Indicator 11 : More private firms in selected cities use ICT and e-services Adjusted RI: More private firms using ICT and e- services in E-Learning Portal Danang. launched Target: 500 (http://multimedia- SMEs well-informed re SME lab.org) with 400 existing services provided managers courses; 268 SMEs by Contact Center; e- trained by took e-learning Learning Portal launched; Value MIC to apply courses; 73 400 ICT and soft-skill 500 managers of quantitative or IT in software developers courses available in e- SMEs trained Qualitative) enterprises; trained; 497 CIOs Learning Portal; 25 1000 SMEs trained; 10% SMEs software developers trained to use using ICT for trained in Requirement Danang e- business; basic IT Management services; 60% Knowledge SMEs use ICT program on TV 3 in business; times/week. and 40% use Danang ICT Transaction Center. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Partially achieved: Restructuring 2013 clarified and quantified 2010 indicator to (incl. % improve link to PDO and measurement. Significant achievements but not linked achievement) closely to indicators and targets. Indicator 12 : New Results Indicator (added by 2013 Restructuring): Number of e-services and socio-economic information available through eGateway (MIC), portal sites (GSO, Hanoi, and Danang) 3 e-services online 1. 3 e-services via MIC; all available via MIC application gateway; radio processes converted transmitter permit; to e-transaction. registr. of plans by Danang: one-gate publishing houses; Value Danang one-gate service services registr. imported quantitative or implemented and implemented, publication Qualitative) operational operational. portfolio of GSO: Portal publishing co's. completed, 2. 1 GSO Portal operational. 3. 1 Hanoi and 1 Hanoi: One-gate Danang one-gate services portal services portal completed. Date achieved 12/31/2012 12/31/2013 12/31/2013 Comments (incl. % Achieved - 100% achievement) New Results Indicator (added by 2013 Restructuring): Indicator 13 : Upgrading ICT infrastructure to support e-government technical capacity in MIC, GSO, MOFA, Hanoi and Danang. Hanoi: HAN5 completed, tested, end-user training. MIC: 83 LANS in provinces; at HQ Completion of 5 and 4 provs, LAN Almost completed in internal ICT and IP phone Value Danang and under networks (LAN): connect all depts. quantitative or implementation in MIC MIC and 12 DICs; MOFA: Data Qualitative) (HQ and DICs), GSO and GSO; MOFA; Center launched; Hanoi Hanoi; Danang. Danang: 95 MAN connections, 32 internet stations, 329 public Wifi access points. GSO: VPN Date achieved 12/31/2012 12/31/2013 12/31/2013 Comments Achieved - 100%: Also achieved: (i) Danang: public Wifi free of charge until (incl. % business pricing model in place; (ii) GSO: All PSOs and 3 ITCs connected via achievement) VPN using MPLS and public internet connection. (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Original Intermediate Outcome Indicators (from PAD): Subproject 1: ICT Leadership and MPT Modernization: Indicator 1 : Workflows and reporting lines streamlined through hardware upgrades and process automation Workflows are 100% of identified Value uncoordinated and workflows and (quantitative na fragmented due to lack of reporting lines or Qualitative) computer systems automated Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Indicator is shown in PAD Results Framework but not ARM. (incl. % Baseline and targets relating to "workflows and reporting lines" are associated in achievement) ARM with IOI 2 - possibly an input error at appraisal. No data available. MPT and DPT architecture in place, including flagship portal allowing Indicator 2 : interaction and transactions between MPT and government, business and citizens. 100% of identified workflows and reporting lines Workflows are automated Value uncoordinated and resulting in (quantitative fragmented due to lack of increased na or Qualitative) integrated computer standardization, systems improved transparency of business processes. Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Baseline and target as presented in PAD ARM and other (incl. % documents seems more consistent with IOI 1, and may be an error. No results achievement) data. Indicator revised by 2010 Restructuring. See IOI 28 below. Indicator 3 : At least 7 appropriate services online by 2010 through the flagship portal e-Services in process of being 7 e-services selected (radio Lengthy transaction times launched Value spectrum, and high costs associated providing services (quantitative publications, with many government to government, or Qualitative) standards for services businesses and integrated citizens communications equipment). Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Negligible achievement: 2010 Restructuring revised this indicator. See IOI 30 (incl. % below. Progress up to 2010 was incipient: selection of appropriate e-services achievement) was initiated. Reduced transaction costs (no. days for response turnaround) on average for 7 e- Indicator 4 : services offered through Flagship Portal e-Services in Evidence of process of being significantly lower selected (radio Lengthy transaction times Value transaction costs spectrum, and high costs associated (quantitative for targeted publications, with many government or Qualitative) government standards for services services (G2C and integrated G2B) communications equipment) Date achieved 08/26/2006 06/30/2011 01/28/2010 Negligible achievement: Expected outputs for IOIs 3 and 4 were dependent on Comments selection of e-services to: (i) establish them online; (ii) assess resulting reduction (incl. % in transaction costs (time). Indicator not in PAD ARM. Broad scope, difficult to achievement) measure. IOs and CIOs from various government agencies and organizations trained in Indicator 5 : ICT issues that are relevant to their position and responsibilities (e.g., technical, legal, knowledge-sharing) Lack of IO and CIO 400 IOs and CIOs awareness and knowledge trained in relevant Value of how ICT can help ICT areas; engage (quantitative na improve administrative in CIO Forum to or Qualitative) processes and govt update ICT services Roadmap Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Target was reduced by 2010 Restructuring to 200 CIOs/IOs (incl. % trained. However, results under revised target not measured/unknown. 2010 achievement) Restructuring revised target again for second stage of project. See IOI 38 below. A CIO Forum established to engage CIOs and MPT staff in the next IT Master Indicator 6 : Plan (2006-2011) and general ICT Roadmap Value CIO Forum; ICT (quantitative Zero na Roadmap or Qualitative) Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: This IOI is included in PAD Results Framework but not ARM, (incl. % and appears to be split from target for IOI 5 above. No results measured. achievement) MPT officials trained in ICT policy and regulatory issues and are knowledgeable Indicator 7 : of best practices in the sector 50% MPT staff trained; increasing Value MPT staff are ICT ICT capacity in 50 MIC staff (quantitative literate; mainly DPT staff respective trained or Qualitative) untrained government departments Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Achieved 100%: IOI target was changed pre-Restructuring to reflect institutional (incl. % transition from MPT to MIC and the target was set at 50 MIC staff trained. achievement) Indicator 8 : Selected DPT staff trained in ICT proficiency and usage Value DPT staff mostly 159 DPT staff (quantitative na untrained in ICT trained or Qualitative) Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: This IOI is shown combined with IOI 7 in the original and the (incl. % 2010 Restructured ARM. No results available/measured. achievement) Improved security around e-services by establishing CERT, CAs and public key Indicator 9 : infrastructure that are inter-operable and in line with international standards Assistance with national CERT, Value Vietnamese telecom CA and public key Draft bidding (quantitative policies do not yet meet infrastructure documents prepared or Qualitative) international standards (PKI) legislation; study of local telecom policies Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Negligible achievement: IOI targets were revised in 2010. See IOI 29 below. achievement) Indicator 10 : Improved Monitoring and Evaluation of the improved ICT strategy Evaluation system No system supporting and annual data Govt to monitor and collection to Value evaluate impact of its support MPT to (quantitative na national ICT stategy on assess impact of or Qualitative) socio-economic ICT on socio- development economic development Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Not achieved: Dropped by the 2010 Restructuring achievement) E-government framework and IT standards in place to guide nationwide e- Indicator 11 : government applications and services Guidelines for e- Govt Architecture e-Govt framework for project cities No guidelines or and IT standards in and agencies issued Value standards to develop e- place to guide in Jan 2009. (quantitative government applications. PIUs to implement Common technical or Qualitative) Lack of IT standards e-Govt standards for inter- nation-wide applications and operability services framework issued in March 2010 Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Achieved 100%: IOI target was clarified in 2010. See IOI 36 below. achievement) Subproject 2: GSO Modernization Indicator 12 : GSO IT architecture upgrade in place, database integration and inter-operability completed Upgrade IT architecture enabling GSO to Low quality accessability Value automate admin. and effectiveness in (quantitative processes; 100% na statistics service delivery or Qualitative) of identified to government and public workflows and reporting lines automated Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: IOI target was not achieved under the first stage of the project but (incl. % was clarified in 2010 and achieved under second stage. See IOI 45 below. achievement) GSO and 50% of the provincial statistical offices connected through Local Area Indicator 13 : Network (LAN), streamlining data dissemination GSO and 50% of 40% (25/63) of Many PSOs not provincial PSOs connected to Value connected to GSO and statistical offices GSO through (quantitative data exchange occurs (PSO) connected, ADSL using or Qualitative) only when there is a improving inter- government funds specific request for data agency data (not IDA credit) transfer Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Partially achieved: IOI target was revised in 2010. See IOI 46 below. achievement) Indicator 14 : Shorter data collection and dissemination cycles Limited data in 4 GSO portal and policy areas intranet launched available to public enabling via Level One Data disseminated in hard Value streamlined website copy format and limited (quantitative electronic data (informational); 10 statistical information or Qualitative) management and days for response to provided online shortened data requests from dissemination govt agencies; 7 times days for data from PSOs Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Partially achieved: Baseline and target not quantified so results not comparable. (incl. % GSO Completion Report IOI was revised in 2010 but not included in the 2010 achievement) revised Results Framework. See IOI 48 below. Enhanced efficiency of data flow and quality of social, economic and poverty- Indicator 15 : related data Value Target associated Similar baseline to IOI 14 See results IOI 48 (quantitative with target for IOI above below or Qualitative) 14 above Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Partially achieved achievement) Indicator 16 : Online access to data collection methods, some data analysis and some raw data GSO Portal and Intranet launched as National Data disseminated in hard Limited data in 4 Value Statistics hub for copy format; limited policy areas (quantitative citizens, statistical information available to public or Qualitative) businesses and provided online via one website govt under free/fee-based arrangements Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Partially achieved: This indicator listed individually in Results Framework but (incl. % combined with/linked to IOIs 14 and 15 in ARM. Revised in 2010. See IOIs 48 achievement) and 49 below. Data demands of government agencies, research institutions and other data users Indicator 17 : met in a shorter period of time. GSO Portal and Data disseminated in hard Intranet launched Limited data in 4 Value copy format; limited enabling policy areas (quantitative statistical information steamlined available to public or Qualitative) provided online electronic data via website management Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Partially achieved: Thsi indicator is listed individually in Resultsisramework but (incl. % combined with/linked to IOIs 14-16 above in ARM, hence duplication of achievement) baseline, targets (partial) and results information. Subproject 3: E-Applications in Hanoi Indicator 18 : City-level government IT architecture in place, including front- and back-end solutions enabling online transactions At 5 key departments and all districts, 100% of workflows and Value No IT architecture to reporting lines (quantitative connect the various na automated, or Qualitative) departments increasing standardization and improving transparency of admin. processes Date achieved 08/26/2011 06/30/2011 01/28/2010 Comments (incl. % Not achieved: Target revised by the 2010 Restructuring. See IOI 51 below. achievement) Selected e-applications online for both G2G and G2B and increased user Indicator 19 : satisfaction Evidence of significantly lower transaction costs Low satsifaction among for targeted govt Value users with respect to services (G2C and (quantitative na many government G2B) eg, land or Qualitative) services registry and population management applications Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Not achieved: Dropped by the 2010 Restructuring achievement) Reduced transaction time and cost through modernization of government offices Indicator 20 : and providing online services Min. 3 additional services available online to public Low satisfaction among and usage Value users with respect to encouraged; (quantitative na many existing evidence of or Qualitative) government services increased user satsifaction and visits to City Portal Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: This IOI appears in PAD Results Framework but in ARM was (incl. % disaggregated into a series of loosely-connected indicators with over-lapping achievement) targets. Dropped by the 2010 Restructuring. Indicator 21 : Increased ICT use by SMEs 15% of SMEs use ICT in business; Value Only 20% of SMEs use SMEs improve (quantitative ICT, and only 5% use it na productivity and or Qualitative) for their business competitiveness through ICT use Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved (in Hanoi): Indicator difficult to measure, target too broad and (incl. % attribution to project difficult. Dropped by 2010 Restructuring. achievement) Indicator 22 : Enhanced competitiveness of SMEs through training and IT services provided at the transaction center 15% of SMEs use ICT in their Value businesses; SMEs (quantitative Same as IOI 21 above improve na or Qualitative) competitiveness and productivity through ICT Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Baseline and target identical to IOI 21. Indicator too broad and (incl. % difficult to measure. Dropped by the 2010 Restructuring. achievement) Government officials and ICT staff in the city government trained in e- Indicator 23 : government and IT issues 50% of IT staff Lack of awareness and and 150 city knowledge of how ICT Value officials trained. can play a role in (quantitative Training increases na improving administrative or Qualitative) ICT capacity in processes and respective govt government services departments Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Indicator revised and clarified by the 2010 Restructuring. See IOI (incl. % 53 below. achievement) Subproject 4: Danang Indicator 24 : City-level government IT architecture in place, including front- and back-end solutions enabling online transactions 100% of identified Danang e-Govt Current workflows are workflows and Enterprise uncoordinated and reporting lines Architecture in Value fragmented due to lack of automated, place. All (quantitative integrated computer improving identified or Qualitative) systems. No IT transparency and workflows and architecture to connect increasing reporting lines the various departments standardization automated. Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Achieved 100%: Front- and back-end solutions applied and transactions now (incl. % online. achievement) Selected e-applications online for both G2G and G2B and increased user Indicator 25 : satisfaction Evidence of increased user Draft bidding Value Low satisfaction among satisfaction and documents ready (quantitative users of many existing visits to City for e-Procurement or Qualitative) government services Portal after content website update and quality check. E- Procurement website launched, linked to City Portal. Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Partially achieved: Indicator ambitious and revised by the 2010 Restructuring. (incl. % See IOI 59 below. Indicator dropped by the 2013 Restructuring. achievement) Reduced transaction time and cost through modernization of government offices Indicator 26 : and providing online services Evidence of significantly lower transaction costs (time) for targeted Value Low satisfaction among govt services Draft bidding (quantitative users of many existing (online business documents prepared or Qualitative) govt services registration and and ready licensing); and, min. 3 additional services online for public Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Negligible achievement: See IOIs 59-61 below achievement) Indicator 27 : Increased ICT use by SMEs SMEs well- informed about existing services at Nationally, only 30% of Value 15% of SMEs use Transaction Center. SMEs use ICT and only (quantitative ICT for their Localized survey 5% for their business or Qualitative) business shows 33% of operations SMEs using ICT and 18% for their business. Date achieved 08/26/2006 06/30/2011 01/28/2010 Exceeded - 120% of target: The nature and function of the Danang Transaction Comments Center was adjusted after 2010 moving from SME focus to more general Public (incl. % Service Center for citizens/SMEs/Officials. For SMEs, focus shifted to ICT achievement) awareness-building Enhanced competitiveness of SMEs through training and IT services provided at Indicator 28 : the Transaction Center SMEs well- informed about ICT Value Only 20% of SMEs use 15% of SMEs use services at (quantitative ICT and only 5% for their ICT in their Transaction Center. or Qualitative) business business 33% of SMEs use ICT and 18% for their business Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Negligible achievement: Baseline and target are identical to IOI 27 and do not (incl. % capture the indicator itself (enhanced competitiveness) which was broad and achievement) difficult to measure. Government officials and ICT staff in the city government trained in e- Indicator 29 : government and IT issues 50% of IT staff City officials have limited and 150 city Value 27 leaders and awareness of ICT use and officials trained; (quantitative officials received management at national training increases or Qualitative) CIO training scale ICT capacity in govt departments Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Partially achieved: Result poorly-measured and not linked to the target. Revised (incl. % by the 2010 Restructuring. See IO 58 below. achievement) Subproject 5: Ho Chi Minh City (HCMC) Indicator 30 : City level IT architecture in place, including front- and back-end solutions enabling online transactions. Current workflows are 100% of identified Value uncoordinated and workflows and (quantitative fragmented due to lack of reporting lines na or Qualitative) integrated computer automated at city systems level Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Restructuring in 2010 dropped this indicator due to HCMC having (incl. % self-financed most activities (zero disbursement of IDA credit) and the HCMC achievement) Subproject itself. Selected e-applications online for both G2G and G2B and increased user Indicator 31 : satisfaction No announcements of e-Procurement Value procurement bids or website launched (quantitative na contracts awarded, for and linked to City or Qualitative) public viewing Portal Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: 2010 Restructuring dropped this indicator (and the HCMC (incl. % Subproject) achievement) Reduced transaction time and cost through modernization of government offices Indicator 32 : and providing online services Evidence of significantly lower Value Low satisfaction among transaction costs (quantitative users of many for targeted na or Qualitative) government services government services (G2C and G2B), eg, driver's license, vehicle registration; and min. 3 additional services available online to public Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments (incl. % Not achieved: This indicator was dropped by the 2010 Restructuring achievement) Indicator 33 : Increased ICT use by SMEs Value Nationally, only 20% of (quantitative SMEs use ICT and only Not defined na or Qualitative) 5% for their business Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: This indicator is not included in the PAD ARM and no targets (incl. % were set. Indicator dropped by the 2010 Restructuring. achievement) Enhanced competitiveness of SMEs through training and IT services provided by Indicator 34 : Transaction Center Value (quantitative Zero Not defined na or Qualitative) Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Ambitious, difficult to measure indicator, not included in PAD (incl. % ARM and no targets set. Dropped by 2010 Restructuring. achievement) Government officials and ICT staff in city government trained in e-government Indicator 35 : and IT issues Value (quantitative Zero Not defined na or Qualitative) Date achieved 08/26/2006 06/30/2011 01/28/2010 Comments Not achieved: Indicator dropped by the 2010 Restructuring, as was HCMC (incl. % Subproject achievement) Post-Restructuring Period (01/28/2010 to 12/31/2013) Subproject 1: ICT Leadership and MIC (formerly MPT) Modernization Indicator 36 : E-Government framework and common IT standards issued for all government agencies E-Govt framework Adjusted IOI: E-govt framework in place to guide Design and in place and TA No guidelines and all provinces and implementatio delivered for this Value framework to develop e- Govt authorities to n of e-govt and IT application (quantitative government applications, implement e-govt framework, standards; or Qualitative) and nationwide lack of IT services; common common IT integrated email standards technical and standards and system feasibility application ICT policies. study; MIC Laws standards for e- Target: E-govt (postal and radio govt issued framework in frequency); place; Information common tech Security Law standards for drafted; MIC e-govt issued; decrees on IT and decrees and Info Security. circulars issued by MIC to business, citizens and gov agencies Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved - 100% achievement) Indicator 37 : ICT policy and implementation capacity through training programs (MIC) Priority ICT Limited capacity to policies and Value formulate policies and implementation (quantitative guide implementation on guidelines na or Qualitative) latest issues in ICT sector. formulated; also, Gap in policy instruments ICT sector M and E framework Date achieved 01/28/2010 06/30/2013 06/30/2013 Comments Not achieved: New indicator added by 2010 Restructuring then dropped by the (incl. % restructuring of 2013. Too broad and its supporting activities not implemented. achievement) Improved ICT awrareness and management capacity through training programs Indicator 38 : (MIC) 497 CIOs and IOs 400 CIOs and trained (277 from IOs trained, of Ministries and 63 Value which 260 DICs, 150 from (quantitative Zero Not defined from Govt state-owned or Qualitative) agencies, 140 enterprises, 70 from state CIOs and IOs from enterprises SMEs) Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Exceeded - 125%: Revised Results Framework of 2010 added this indicator (incl. % without providing monitoring details in revised ARM; 2013 Restructuring achievement) assigned specific, measurable targets Indicator 39 : Improved MOFA e-visa services under MIC's oversight Average processing time Value Taking 5 or more days to for visa (quantitative na obtain a visa application or Qualitative) reduced by at least one day. Date achieved 01/28/2010 06/30/2013 06/30/2013 Comments Dropped: This indicator was added by the 2010 Restructuring and dropped by (incl. % 2013 Restructuring. Visa targets were combined with and reported in IOI 42 achievement) below. Indicator 40 : Enhanced MIC and DICs' communications through ICT infrastructure upgrade Adjusted IOI: Completed ICT infrast. upgrade to MIC installed audio enhance and video communicatio conferencing, smart Improved ICT ns between building communic. infrastructure for MIC, DICs Workflows uncoordinated between MIC and 4 MIC (conferencing and MOFA. and fragmented due to agencies (MIC Value facilities, portal Target: lack of integrated Regional Offices (quantitative infrastructure); and Improved ICT computer systems. Danang and or Qualitative) min. 5 DICs infrastr. MIC Limited access to ICT HCMC, telecom equipped with ICT and MOFA infrastructure. authority in Hanoi infrastructure and (conferencing and MOFA). E- Portal facilities, Gateway (MIC portal infrastr., Portal) completed data center); and operational. min. 12 DICs with ICT infrastr., VC and LAN Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Partially achieved achievement) Indicator 41 : Core MIC services for quality and security assurance Adjusted IOI: Upgrade of Root Quality and Certification security MIC: By end-2013, Authority (CA) to assurance for all systems (Root perform advanced ICT services CA, Test Center) PKI services; in Vietnam were 100% systems to issue through core completed and end- Value Vietnamese telecom digital certificates MIC services. users trained. (quantitative policies do not yet meet to citizens and Target: Systems for digital or Qualitative) international standards business for digital Upgrade CA; certificates and signature; Test systems for signature Center established digital successfully to evaluate IT certificates developed and products for e- and signature; transferred govt. Test Center established. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Achieved 100% (incl. % achievement) Indicator 42 : Three licensing e-services offered to businesses and citizens available online Adjusted IOI: Time savings of new 3 licensing services systems (3 launched in May licensing e- 2013 including visa services and online system (95 Lengthy transaction times Value 3 licensing visa online visa offices and high formal and (quantitative services available system). worldwide and informal costs associated or Qualitative) online Target: application process with many govt services Transaction piloted and time for visa launched, time processing reduced to 2.5 reduced (by days). min. visits per applicant). Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved 100% achievement) Indicator 43 : 200 CIOs and IOs trained Lack of CIO and IO 497 CIOs and IOs awareness and knowledge Target trained (277 from Value of how ICT can play a 200 CIOs and IOs retained by Ministries, 63 (quantitative role in improving trained 2013 DICs, 220 from or Qualitative) administrative processes Restructuring State-owned and govt services Enterprises Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Exceeded: 249% This indicator is shown in the Results Framework and ARM. (incl. % Overlaps with IOI 44 achievement) 200 MIC staff trained on policy formulation, capacity-building and Indicator 44 : implementation guidelines for IT and telecommunications sectors Target Value Staff in MPT well-versed 200 MIC staff retained by Exactly as IOI 43 (quantitative in ICT literacy, but DPT trained (as above) 2013 above or Qualitative) staff mainly untrained Restructuring Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Exceeded - 249% This indicator is shown in both the Results Framework and (incl. % ARM. Duplication error with IOI 43 achievement) Indicator 45 : Subproject 2: GSO Modernization Adjusted IOI: GSO EA in place; GSO architecture Low quality, accessability Design and all IT initiatives at Value developed and and effectiveness in implementatio GSO must follow (quantitative 80% of statistics service delivery n of GSO IT standard; GSO EA or Qualitative) departments to govt and public architecture. artifacts captured in included Target: GSO System Architect EA in place; Software; all GSO EA artifacts investments follow captured in EA; centralized and repository coordinated IT tool; services via SSIC centralized and SHS; IT and priorities based on coordinated IT project activities services; effective IT investment and priority- setting. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved 100% achievement) Improved data transfer through LAN connection between three GSO IT Centers Indicator 46 : and Provincial Statistical Offices (PSOs). Adjusted IOI: IT infrastr. upgrade completed All PSOs and 3 (improved ITCs connected data transfer with GSO HQ via via LAN VPN using MPLS 63% of PSOs between GSO, and public internet; (40/63) and 3 IT 3 IT Centers, new email system Many PSOs not Centers connected PSOs). Target: Value for secure connected to GSO and to GSO via VPN Commun. (quantitative document data exchange occurs network solution, between GSO or Qualitative) exchanges; data only upon special request improving data and PSOs transfer improved transfer between enhanced: GSO and PSOs; Govt agencies 100% PSOs network and 3 IT connectivity for Centers GSO Enterprise connected to Applications GSO via wide area VPN. See value achieved for other. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved 100% achievement) Increased efficiency of data management and report generation through database Indicator 47 : integration (GSO) Value Limited no. of standalone Target expanded: Adjusted IOI: 3 of 9 sector (quantitative databases some of which Over 60 databases Time saving of databases or Qualitative) contain redundant integrated via data new data developed (pop, information. No data warehouse; data management SME and BDS); standardization or mining and and report data standard. and integration methodology. management generation via integ. methodology Monthly reports significantly database developed; pop generated manually. improved via data integration. reports extraction Target: automatically applications; and, Common generated; SSIC automated report SSIC system implemented at generation. to support GSO HQ and 3 GSO surveys ITCs. End user and data training completed. collection; 2 GSO depts. pilot SSIC; 6 depts move to SSIC; min. 30- 50% reduction turnaround time. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Partially achieved: No data available on savings in turnaround time. achievement) Indicator 48 : Shortened survey data collection and processing Adjusted IOI: Time saving of Data collection, survey data processing for 2009 collection and Census 8 months processing. (33% reduction); Target: Form Forms Scanning Scanning Infra. used for Previous census data Shorten data Infrastr. to Value 2011 Census (50% collection and processing collection and support large- (quantitative better o/all took 12 months (for data processing to 12 scale census or Qualitative) processing time, 2 capture only) months projects with years to 1); SSIC 50% time package tested for improvement; user acceptance; 30-50% SHS finalized; CPI improvement survey time reduced in data entry 33%. for IT-enabled processing. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved 100% achievement) Improved transparency through wider access to data and shortened turnaround Indicator 49 : time for data requests through GSO Portal Value Statisticians working on Shorter Adjusted IOI: GSO: All GSO (quantitative standalone PCs; dissemination Number of statistical data or Qualitative) dissemination through cycles from survey domains integrated hard copy; limited increased databases with GSO Portal statistical information interaction through available and basic statistical online GSO Portal and through GSO data is provided to intranet; and GSO Portal. Target: public free of Portal as National Up to 60% of charge. Fee-based Statistics Hub for databases system under citizens, business integrated consideration. 50% and govt (free and through data of GSO data now fee-based) warehouse and integrated in Stat. available on Data Warehouse. GSO Portal Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Substantial achievement: 84% This indicator is not in the 2010 revised Results (incl. % Framework but is shown in ARM and was formalized by the 2013 Restructuring. achievement) Improved capacity of staff in performing streamlined and ICT-enabled GSO data Indicator 50 : processes 122 CIOs trained and certified; 83 Adjusted GSO staff and IT target: experts from 3 Number of centers trained in staff trained to Lack of awareness and Network Admin; increase Value knowledge of how ICT 670 GSO and PSO 395 GSO and 63 knowledge on (quantitative can improve staff trained and/or PSO staff and ICT issues or Qualitative) administrative processes accredited statisticians trained relevant to and govt services. in MS Office; 27 IT further experts in 3 IT modernization Centers trained in of GSO SQL, Share Point, functions New Windows Server. Total 690 Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Exceeded - 103% achievement) Subproject 3: E-Applications in Hanoi Indicator 51 : City-level IT architecture in place Revised target: EA Master Plan Design EA designed for 3 Master Plan levels of Govt for 3 levels of (Dept, Ward and Value 57 Departments, No IT architecture to Govt; and Commune); City- (quantitative Districts aligned to connect city departments detailed EAs level IT architecture or Qualitative) IT architecture for designed; Aligned Departments, architecture Wards and designed for 57 Communes; Departments develop aligned IT architecture for 57 Departments. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Achieved - 100% Continued by Restructuring of 2013 but with more ambitious (incl. % targets. Also, the 2013 Restructuring split Hanoi and Danang into 2 components. achievement) Indicator 52 : Better access to government services through the single City Portal Adjusted IOI: Data Center for No. of public City Portal services completed and available software installed; One-stop-shop through City No one-stop-shop 18 dept portals and Portal Portal. Target: available for city 16 applics. Value infrastructure in One-stop shop government services, integrated with (quantitative place for access to software for which are provided and Hanoi City Data or Qualitative) e-services via govt agencies processes through various Center; One-stop many departments integrated in departments and locations software for Govt and districts Data Center; agencies integrated integration of in Data center; 57 57 portals and dept. portals and websites of websites integrated. Departments. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Achieved - 100% Access to Govt information now easier, more reliable. (incl. % Also,public services integrated via Internet (20 at 2nd level, 8 at 3rd level, 2 at achievement) 4th level). Improved ICT awareness and management capacity through training programs Indicator 53 : (Hanoi) Adjusted IOI: No. of city officials Lack of information 57 leaders receive (Hanoi only) officer and CIO CIO training; 500 take CIO Value awareness and knowledge SME managers training 30 leaders trained (quantitative of how ICT can play a trained in courses. in Singapore (2011) or Qualitative) role in improving application of IT Target: 340 administrative processes to their enterprises leaders and and govt services key officials receive CIO training Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Negligible achievement (with IDA funds) of 9%: Training for SMEs was (incl. % originally contemplated using IDA credit but was implemented with local funds. achievement) Over 100 SMEs trained in CIO. Indicator 54 : Improved awareness and use of ICT among private enterprises including SMEs Value Only 20% of SMEs use Not defined na (quantitative ICT, and only 5% for or Qualitative) their business (nationally) Date achieved 01/28/2010 06/30/2013 12/31/2013 Comments Results unknown, indicator dropped: This indicator was included in the 2010 (incl. % revised Results Framework both as an Outcome Indicator and - in similar form - achievement) an IOI but the ARM provided no targets for latter and it was dropped. Reduction of time required for one transaction of selected government services Indicator 55 : through City Portal compared to traditional way Transaction time through one-gate Transaction time system reduced by Target Value Only 20% of SMEs use through one-gate 50% compared to retained by (quantitative ICT and only 5% for their system reduced by traditional way 2013 or Qualitative) business 50% compared to (eg, 5 days for Restructuring traditional mode. business registration) Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Achieved - 100% This IOI was added in 2013 and monitored by ISRs but was (incl. % not included in the 2013 adjusted Results Framework achievement) Subproject 4: Danang Indicator 56 : City-level Government and IT architecture in place Adjusted IOI: Design and implement. of city-level IT Danang e-Govt EA architecture. in place; 27 Depts, Target: 91 8 Districts, 56 authority units Wards, Communes aligned; e- have LAN and Govt EA Internet connection; Value No IT architecture to 92 authority units framework MAN operational (quantitative connect various are aligned with IT and model for providing Fiber or Qualitative) departments architecture Danang Optic Cable approved by connection between PPC; report on 97 state agencies; strategy, 30 addit. depts and roadmap, arch. agencies also standard and equipped. Total 121 prototype of units achieved. Danang City prep. and approved. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Exceeded - 133% Indicators from PAD were sustained by 2010 Restructuring (incl. % then clarified and adjusted in 2013. achievement) Indicator 57 : Better access to Government services through the City Portal Adjusted IOI: Better access to govt Contact Center in services via place (881 888) different providing info on channels. public services and Target: Public socioeconomic 80% of the 18 Wifi covers issues; MAN City government services identified services public areas of Value connecting 97 are provided and provided through city; City (quantitative depts., agencies, processed through various City Portal are Portal with or Qualitative) communes, wards; departments and locations active and used by links to other 32 Internet Stations citizens depts, districts in communes, etc.; Public wards; 10 inform. Service Info. kiosks for tourists; Portal and 329 free public Public Contact Wifi access points. Center launched. 10 tourist kiosks Date achieved 01/28/2010 06/30/2013 12/31/2013 12/13/2013 Comments Achieved - 100% Indicator revised by the 2010 Restructuring then adjusted in (incl. % 2013. Danang has commissioned additional Wifi access points and Wifi is achievement) available free of charge until a business pricing model is put in place. Improved ICT awareness andmanagement of city officials through training Indicator 58 : programs Adjusted IOI: No. of city 27 leaders, officials officials take received CIO training and IT training; 8 IT training technicians did IT courses. courses; 102 target: Min. City officials have limited At least 150 participated in Value 27 leaders awareness of ICT use and leaders receive foreign study tours; (quantitative receive CIO management at national CIO training and 141 officials did IT or Qualitative) training; min. scale IT training courses training (CCNA, 30 leaders MCSA, Oracle DB, trained in ICT Database Strategic Networking); 400 Management; officials did end- min. 400 user training. officials trained in IT Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Achieved - 100% A similar indicator was included in PAD and revised by 2010 (incl. % Restructuring (see above). Included in the adjusted 2013 Results Framework. achievement) Improved transparency and reduced transaction costs (i.e., time) for public Indicator 59 : procurement Value Few announcements of e-Procurement na (quantitative procurement bids or system launched or Qualitative) contracts awarded and integrated into available for public view. City Portal; 60% users satisfied with e-Procurement system; 25% of procurement bids processed through this system Date achieved 01/28/2010 06/30/2013 12/31/2013 Comments Indicator dropped: Indicator was created at appraisal, continued by the 2010 (incl. % Restructuring but dropped in 2013 because financed by other sources (non- achievement) project). Improved transparency and reduced transaction costs (time) for business Indicator 60 : registration and investment licenses Adjusted target: e- Business Business Registration Registration System system e-Business and Investment launched and Registration Project integrated into Value Business registration and system launched Management City Portal; (quantitative license processing done and integrated into System completed 70% of users or Qualitative) offline City Portal; 70% development phase satisfied; 80% of users satisfied and launched in of investors with this system. January 2014. No applying for user surveys their conducted yet. investment licenses online. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Partially achieved: Due to the January 2014 launch, there was inadequate time to (incl. % conduct user satisfaction survey or to assess degree of online usage for achievement) investment licensing. Improved transparency and reduced transaction costs (time) for vehicle Indicator 61 : registration and driver's license. Adjusted IOI: Vehicle Improved registration system transparency launched and Driver's license and reduced integrated into system developed transaction City Portal; and completed; Value costs (time) Driver's license Driver's License deployment for (quantitative for driver's processing done offline system launched commissioning and or Qualitative) license. and integrated into user acceptance test Target: City Portal; 70% conducted. System Driver's of users satisfied fully operational. License with these new system systems launched and integrated to City Portal; 70% users satisfied with driver's license system. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Comments Partially achieved: Indicator added by the 2010 Restructuring and adjusted by (incl. % the 2013 Restructuring. Inadequate time since launch to assess user satisfaction achievement) with online licensing system. Indicator 62 : Improved awareness and use of ICT among private enterprises including SMEs Adjusted IOI: No. participants from SMEs take ICT training to enhance ICT 1000 participants SMEs now well- skills for from SMEs trained informed of e- business in e- in e-commerce, e- services via Contact commerce, e- procurement, and Center; e-Learning Value Only 20% of SMEs use procurement applying e- Portal launched and (quantitative ICT and 5% do so for and e-services. services; 60% of 400 ICT courses or Qualitative) their business. Target: 1000 SMEs use ICT in available incl. 15 SME business; 40% of modules for SMEs; participants SMEs use ICT 73 software trained in Transaction Center developers trained those 3 services; all SMEs supported via Contact Center. See Comments. Date achieved 01/28/2010 06/30/2013 12/31/2013 12/31/2013 Exceeded: Revised target also included 400 courses in ICT soft-skills on e- Comments Learning Portal and min. 75 SME software developers did software engineering (incl. % training. Other results: 50 SMEs did e-Govt Platform courses; basic IT achievement) knowledge progs on TV weekly. New indicators created by 2013 Restructuring (Danang only): Indicator 63 : Improved ICT infrastructure of the City Government Danang MAN Completed: Metro with fiber optic Area Network; connection to min. public Wifi with Value No significant ICT 90 Depts., 165 public free (quantitative infrastructure developed agencies; Data access points; or Qualitative) by the City Government Center; public additional Wifi Wifi; Training and points completed; Applied Research Data Center; e- Center; Danang e- Learning Portal Learning Portal; e- operational. Govt Framework; all Depts, wards. agencies, communes with end-user IT equipment. Date achieved 06/20/2013 12/31/2013 12/31/2013 Comments Achieved - 100% Due to overall strong performance, Danang received over (incl. % US$10.0 m. in additional IDA funding through reallocation and was able to achievement) achieve beyond expectations. Improved transparency and reduced transaction costs (time) for Construction Indicator 64 : Permits Construction Permit System launched and System completed Value integrated into development and (quantitative Zero (not surveyed) City Portal, 30% commissioning or Qualitative) of applications phases, launched online; 70% of January 2014 users satisfied with the system Date achieved 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Partially achieved: Insufficient time since launch to conduct user survey. achievement) Indicator 65 : Enhance efficiency in Public Transport Management System was Public Transport commissioned and Management 100% of buses are Value launched; and, equipped with GIS, (quantitative Zero 100% of buses are managed and or Qualitative) managed centrally monitored centrally and based on GPS by DOT, and based and GIS Map. on GIS Map. Date achieved 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved - 100% achievement) Indicator 66 : Enhance efficiency of Water Resource Quality Management Water Monitoring Water Monitoring System launched; System operational; raw water raw water Value monitored and monitored and (quantitative Zero controlled 24/7 on controlled 24/7 on or Qualitative) criteria before criteria (color, treatment and smell, taste, processing stage muddiness, residual chlorine, pH, and Ammonia, Fluoride and Chloride content, and coliform) before treatment and processing. Date achieved 06/30/2013 12/31/2013 12/31/2013 Comments Achieved - 100% Social and health benefits were not studied but expected to be (incl. % substantial based on global experience. achievement) Indicator 67 : More efficient government administration 90 Depts, agencies 7 districts, 56 got video phones; Wards use e- VC service District, e-Wards established; system; 100% use Business Process one-stop shop at database developed District, Ward Value and operational; e- level; Business (quantitative Zero Wards and e- Process Database or Qualitative) Districts for public admin commissioned; processes; MIS 100% one-stop launched; min. 90 shops connected to Depts and MAN and agencies establish transferring data to a VC Committee. Dept Home Affairs. Date achieved 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved - 100% achievement) Indicator 68 : Enhance and develop the ICT Human Resources in Danang City Danang ICT HR ICT HR Devt Devt Master Plan Master Plan approved; e- approved by PPC; Learning portal e-Learning Portal launched; 400 launched, 400 soft-skills courses courses; 75 in e-Learning software devel. Value Portal; min 75 trained in Software (quantitative Zero software Engineering, Proj. or Qualitative) developers (pub. Manag. and and private) get Testing; 27 leaders Software Eng. did CIO training; training; min. 27 25 engineers, Integ. leaders get CIO Circuit Design; 400 training; min. 400 officials trained in officials trained in IT. IT Date achieved 06/30/2013 12/31/2013 12/31/2013 Comments (incl. % Achieved - 100% achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 10/30/2006 Satisfactory Moderately Satisfactory 0.00 2 07/21/2007 Satisfactory Satisfactory 3.83 3 02/09/2008 Moderately Satisfactory Moderately Satisfactory 3.83 4 09/10/2008 Moderately Satisfactory Moderately Satisfactory 4.50 Moderately Moderately 5 06/11/2009 5.40 Unsatisfactory Unsatisfactory Moderately Moderately 6 04/13/2010 8.04 Unsatisfactory Unsatisfactory 7 05/02/2011 Moderately Satisfactory Moderately Satisfactory 13.57 Moderately Moderately 8 02/21/2012 19.78 Unsatisfactory Unsatisfactory Moderately Moderately 9 08/05/2012 24.23 Unsatisfactory Unsatisfactory Moderately Moderately 10 01/02/2013 30.75 Unsatisfactory Unsatisfactory Moderately 11 09/28/2013 Moderately Satisfactory 55.99 Unsatisfactory Moderately 12 01/13/2014 Moderately Satisfactory 63.16 Unsatisfactory H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Level One Restructuring revised the PDO and Outcome Indicators to focus more 01/28/2010 Y MU MU 7.35 narrowly on project activities funded by IDA. Hanoi Subproject dropped and components changed. Level two Restructuring extended closing date by two 06/29/2011 MS MS 15.54 years to allow Level One changes to take hold and ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions improve progress. Level two Restructuring extended closing date 6 months, adjusted Results Framework, reallocated US$10.6 m and 06/20/2013 MU MU 45.18 cancelled US$10.3 m. Purpose was to allow time to complete all procurement and achieve PDO. If PDO and/or Key Outcome Targets were formally revised (approved by the original approving body) enter ratings below: Outcome Ratings Against Original PDO/Targets Unsatisfactory Against Formally Revised PDO/Targets Moderately Unsatisfactory Overall (weighted) rating Moderately Unsatisfactory I. Disbursement Profile 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1.1.1 Country context and national commitment: Vietnam, like many developing countries, recognizes the value and potential of information and communication technology (ICT) to enhance competitiveness, facilitate economic diversification and increase productivity, and to improve institutional capacity and efficiency especially in government. At the time of appraisal in 2005, the Government of Vietnam (GOV) had repeatedly identified ICT as a top priority, issued high-level decisions and formulated strategies, master plans and decrees to that effect. These included the National ICT Development Strategy 2006-2010, the e-Government Development Master Plan to 2010, the State Administration Modernization and Computerization Program and the Prime Minister’s Decision 169 on administrative simplification. The Ministry of Posts and Telematics (MPT) was to lead implementation of the National ICT Strategy and E-Government Master Plan under the guidance of a National Information Technology Steering Committee. Key targets of the Master Plan included: extensive diffusion of ICT throughout the national economy and society to increase ICT’s share of GDP; establishing a national information and communications network; and, comprehensive ICT skills development. 1.1.2 Important efforts to upgrade ICT infrastructure flowed directly from this enabling framework including: introducing information systems into selected government departments and piloting online services; providing incentives to the ICT industry; and developing ICT skills and improving awareness of ICT in government and to some extent, the private sector. Vietnamese authorities expected that ICT development would be an instrument to facilitate service delivery and a potential source of growth. References to ICT development featured prominently in GOV’s Socio-economic Development Plan (2006-2010). Government had also committed under international trade agreements (including the Vietnam/US bilateral trade agreement) to open the telecommunications sector to outside investment and to increase competition. Substantial progress had been already made in moving towards international norms and best practice. 1.1.3 Constraints and challenges: Government nevertheless faced tough constraints to e- readiness at the time of project appraisal: the high cost of and uneven access to communications infrastructure; a shortage of skilled human resources including in ICT-related disciplines; and limited diffusion of ICT within government, the business sector and citizenry. Achieving the new Master Plan’s targets was challenged further by the need to strengthen the management, coordination, coherence and result-orientation of on-going programs. This included establishing realistic indicators and benchmarks and developing successful demonstration models or applications which could be scaled up over time. 1.1.4 Rationale for Bank involvement: GOV sought Bank support for ICT development for the following reasons: (i) the Bank’s active, multi-country engagement in telecommunications reform, e-government and ICT development throughout the developing world including East Asia and the Pacific region, and comparative advantage in promoting best practice experiences in ICT projects and policy formulation; (ii) established dialogue with GOV on telecommunications and ICT policy issues and knowledge of main stakeholders; (iii) ability to coordinate development partners working in ICT to prevent overlap and duplication; (iv) understanding of the ways ICT could be integrated in other sectors, thus contributing more broadly to growth and poverty reduction; and, (v) capacity to provide long-term, large-scale financing. 1 1.1.5 Contribution to higher-level objectives: The Project supported two key pillars of the Bank’s Country Assistance Strategy (CAS) FY03-FY06: (i) high growth through transition to a market economy, by developing policies for the ICT sector to generate employment, exports and domestic economic efficiency in Vietnam, and through training to build a computer and internet- literate small and medium enterprise (SME) sector more able to deliver growth and engage in international trade and competition; and (ii) adoption of a modern public administration, legal and governance system. Promoting efficiency and transparency of government operations via portals, online applications for business and land registration, e-procurement, and data collection and dissemination would improve government decision-making processes, reduce transaction costs, improve quality and standardization of government documents and limit the scope for corruption. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 1.2.1 The PDO as stated in the PAD and Development Credit Agreement sought: (i) to accelerate the use of ICT and support institutional development in order to increase the efficiency and effectiveness of public administration, increase transparency and availability of information, and improve public services delivery to stakeholders; and (ii) to foster private sector development in particular by facilitating the increased adoption of ICT by small and medium enterprises. 1.2.2 The table shows the alignment between the original PDO and Results Indicators: PDO Project Outcome Indicators 1. Accelerate the use of ICT and support -Increase of 2% in share of ICT sector in GDP institutional development to increase -Increase in ICT capacity and knowledge in respective government transparency and availability of departments information, and improve public services -Increase in user satisfaction for e-government services and visits to delivery to stakeholders. City-portal. -Increase in standardization and improvement in transparency of administrative processes -Significantly lowered transaction costs for targeted government services (G2C and G2B) 2. Foster private sector development in -15% use ICT for business operations in 2010 and improve particular by facilitating the increased productivity adoption of ICT by SMEs -Domestic IT service sector grows by an average 30% per year 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 1.3.1 A Board-approved Level One restructuring in January 2010 revised the PDO and Outcome Indicators to focus more narrowly on project activities funded by IDA, assisting the Recipient: (i) to support the MIC’s institutional development and technical and regulatory capacity for ICT sector oversight and to increase efficiency and transparency of public service delivery in MIC, GSO, MOFA, 1 Hanoi and Danang; and (ii) foster private sector development by increasing access to business-related government services and raising awareness and usage of ICT in Hanoi and Danang. 1.3.2 The table below shows alignment between the restructured PDO and Results Indicators as at January 2010. A second (Level Two) Restructuring in June 2013 adjusted the Results Framework including by adding two new Results Indicators. For convenience, these are shown in the table below and discussed in Section 2.3 and Annex 2. 1 Respectively: Ministry of Information and Communication (MIC); General Statistics Office (GSO); Ministry of Financial Affairs (MOFA). 2 Restructured PDO Revised Indicators 1. Support MIC’s institutional development -Improved formulation and implementation of ICT policies and and technical and regulatory capacity for common standards ICT sector oversight and to increase -Reduced transaction costs, increased transparency of and users’ efficiency and transparency of public satisfaction with selected government services through e- service delivery in MIC, GSO, MOFA, applications** Hanoi and Danang. Added by 2013 Restructuring: -Number of e-services and socio-economic information available through e-Gateway (MIC), portal sites (GSO, Hanoi, Danang) -Upgrading ICT infrastructure to support e-government technical capacity in MIC, GSO, MOFA, Hanoi and Danang 2. Foster private sector development by -G2B services made available online. increasing access to business-related -More private firms in selected cities use ICT government services and raising awareness and usage of ICT in Hanoi and Danang. **Also applicable to PDO #2 1.3.3 Rationale for changes: The partially adaptive, partially corrective Level One restructuring was requested by the State Bank of Vietnam for the following reasons: 2 • HCMC and Hanoi Subprojects had adequate funds from own sources to conduct activities similar to the Project. In the case of HCMC, the Project Implementation Unit (PIU) had tried hard to absorb the city’s US$3-4 m. budget allocation but could not use the additional IDA resources. HCMC Subproject disbursement was virtually zero. • Demand for IDA funds in GSO and Danang to finance their ICT development plans was increasing. Both PIUs were strong performers and reallocation of funds to their Subprojects was thought likely to ensure timely completion of project activities. • Implementation capacity was weak in the Hanoi and MIC PIUs (and PCU), hindering progress in the first half of the Project. See also Section 2.2. 1.4 Main Beneficiaries: 1.4.1 Project beneficiaries were not described explicitly in the PAD but by inference were: government institutions at both the central and decentralized levels, and especially those directly involved in project implementation; small and medium-sized business enterprises; and, the general public/citizens through improved access to e-services. Project restructuring in 2010 did not change the identity of expected beneficiaries but did expand the estimated numbers likely to benefit from increased public access to e-services (see Section 2.2). 1.5 Original Components (as approved) 1.5.1 The Project comprised five Subprojects in three areas: (i) ICT leadership and modernization of the MPT; (ii) modernization of the GSO; and, (iii) e-applications for the municipalities of Hanoi, Danang, and Ho Chi Minh City. The five subprojects covered a total 14 “components” or sets of activities. See Annex 2. Subproject 1: ICT Leadership and MPT (later MIC) Modernization (total estimated cost US$21.7 m. of which IDA US$20.1 m., 93%) financed capacity-building for MPT for the development, implementation and oversight of government’s ICT strategy and policies; formulation of telecommunications and ICT policy; and, improving MPT’s operating efficiency and effectiveness. The Subproject had six components/activity sets: (i) development of e- 2 Bank management was ambivalent about pursuing a first order restructuring based on concerns about the time and effort needed for its approval, especially in Vietnam, potentially adding to existing implementation delays. 3 government capability in MPT; (ii) informational interaction and transactional portal design and activation; (iii) establishing a training program, and the design and implementation of curricula for the Chief Information Officers (CIO); (iv) capacity-building for ICT policy and regulatory development; (v) development of a legal, regulatory and institutional framework for IT and telecommunications; and (vi) development of a national IT architecture and inter-operability framework for e-government. See Annex 2. Subproject 2: GSO Modernization (total estimated cost US$13.77 m of which IDA 100%) supported implementation of the strategic plan for modernization and capacity-building of GSO. The quality, transparency and timeliness of socio-economic and poverty data/information for government agencies, research institutions and other data users were expected to improve, modernizing the foundation for policy, planning and decision-making. Systems would be upgraded and the timeliness, analytical quality and dissemination of data/statistical information would improve. The Subproject had three components/activity sets: (i) ICT infrastructure and process upgrade; (ii) data flow and enhanced statistical analysis through ICT; and (iii) development/expansion of ICT human resources and awareness. See Annex 2. Subprojects 3, 4 and 5: E-Government in Hanoi, Danang and Ho Chi Minh City (total estimated cost for Hanoi US$31.3 m/IDA US$28.06 m (89.6%); for Danang US$17.33 m./IDA US$16.42 m. (95%); and for Ho Chi Minh City US$10.61 m./IDA US$8.93 m. (84%)), sought to improve the ICT environment in these cities by providing the foundation for deployment of e- government in district offices and departments. E-government services, e-applications for business, and ICT skills would improve government accountability, effectiveness, transparency, public and business access to information, citizen and business interaction and public administrative reform efforts. SME competitiveness was expected to improve with spillover effects on trade opportunities, employment and income. The Subproject had six components/activity sets: (i) development of ICT strategy, policies and architecture and upgrade of ICT infrastructure, equipment and systems for these cities; (ii) assessment of business processes and development and implementation of workflow solutions; (iii) development and expansion of electronic gateways and city portals; (iv) implementation of e-applications in selected public services, including geographic information systems; (v) support for the uptake of ICT by SMEs through ICT transaction centers; and (vi) development of ICT human resources and awareness-raising. See Annex 2. Subproject 6: Project Coordination (total cost US$1.5 m. of which IDA 100%) financed a Project Coordination Unit (PCU) within MPT (later MIC) - which was a separate unit from the MIC PIU - for overall coordination of project monitoring and reporting, and knowledge-sharing across subprojects. Implementation would be decentralized; each Subproject would have its own PIU (in MIC’s case, as well as the PCU) and Special Account. See Annex 2. 1.6 Revised Components 1.6.1 The 2010 Restructuring revised several Subprojects/components (see Annex 2): • Ho Chi Minh City (HCMC): Subproject was cancelled (see 1.3.3). • MIC: Activities added: Test Center for IT product evaluation; consultancies to prepare plans for e-document interchange and integrated e-mail systems; and installation of an intelligent building system and integrated communications. The planned National Computer Emergency Readiness Team (CERT) was removed (government-funded). • MOFA under MIC: Data Center and establishment of online visa services added. 4 • GSO: Planned Information Warehouse would add a data warehouse and information dissemination portal. New components added (US$6.7 m increase in credit amount by reallocation): equipment for data collection/processing for the 2009 Population Census; and, GIS application support mapping administration field database. • Danang: Activities were expanded (US$10m increase in credit amount by reallocation): Data Center added to the Transaction Center, and Danang’s ICT infrastructure upgrade program expanded to include 30 departments, seven districts, and 56 communes and wards. • Hanoi: IT infrastructure upgrade was removed, the number of e-services was reduced, and a Data Center was added. 1.7 Other significant changes 1.7.1 Restructuring and amendments to the DCA: • Level Two restructuring of June 29, 2011 approved by senior Bank management extended the closing date by two years to June 30, 2013. • Level Two restructuring of June 20, 2013, approved by Bank management on an exceptional basis (IP and DO ratings were then MU, and prior extensions totaled 24 months), extended the closing date six months to December 31, 2013; adjusted the Results Framework (see Section 2.3); reallocated US$10.6 m. and cancelled US$10.3 m. of the IDA Credit (and US$4.6 m. of counterpart funds) representing large procurement packages either already financed by government with its own funds or unlikely to be completed by the closing date. The justification was to allow time to: (i) complete major contracts signed by December 31, 2012; (ii) stabilize and consolidate the ICT systems already established; and (iii) support critical turnkey packages signed before December 31, 2012 to ensure that software applications and IT equipment functioned properly. 3 • Development Credit Agreement (DCA) amended in 2008 to reflect an agreement during negotiations that IDA would finance the reasonable costs of incremental expenditures by the PCU in coordinating the project. 4 US$80,000 was allocated for this function. • DCA amended in February 2010 to substitute the Ministry of Information and Communication (MIC) as lead executing agency in place of the Ministry of Posts and Telematics (MPT), and Department of Information and Communication (DIC) for Department of Posts and Telematics (DPT). 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 2.1.1 Preparation and background analysis: The PAD reflects a comprehensive effort to design an ICT in the mainstream of best practice. External specialist sources including academic 3 The rationale included: (i) experience with other ICT projects (difficult to launch and implement); (ii) Vietnam country context (slow project implementation); (iii) importance of ICT to the country’s transparency agenda which the Bank was fostering; (iv) progress on several complex ICT contracting processes and need for time to ensure that newly- installed technologies worked; (v) country fiscal constraints, i.e., unfinished works were unlikely to be finished by the client with own-funds; and (vi) emergence of important project results and impact. Critically, certain pending procurement packages were linked directly to achievement of the PDO. 4 The Bank Task Team believed that the O&M costs of the PIUs in Hanoi, Da Nang and Ho Chi Minh City, and at the GSO, should be borne by counterparts to ensure ownership of project activities and the longer-term financial sustainability of project activities. Incremental operating costs of the Project Coordination Unit however, were strictly project-related, would not have been incurred absent the project, and would not continue beyond closing. 5 were consulted. The internal review process was thorough, and wide-ranging peer input was sought. Concerns were expressed internally about the following: project scope and complexity; institutional arrangements, especially the cost and efficiency of using five separate implementation units; M&E and the quality and relevance of the Results Framework especially impact indicators; overall coherence of the proposed intervention and linkage between the five Sub-projects; inadequate connection between proposed sustainability factors and what was likely to work; and, weakness in the underlying analytical justification for key activities including SME involvement. Evidence suggests that while the project team understood these concerns, project design responded only partially and the internal clearance process acquiesced. 2.1.2 Lessons: Global experience with ICT operations showed that the primary drivers of success were strong country ownership and sustained commitment to implementation. Thus the consistency of project design with government’s National ICT Strategy (2006-10) and other high- level policy documents was equated with full ownership by government. Global lessons in organization, leadership models, personnel and procedures in setting up telecommunications and ICT regulatory bodies shaped design, while experiences balancing progressive change with internal resistance and political issues were used to create a realistic roadmap for policy change. For e-applications development, the Project heeded experiences in e-government projects in the developed and developing world, and World Bank/other case studies. The value of picking high payoff focus areas such as the e-registration of businesses and e-procurement was also studied and incorporated (but the latter was later dropped). Lessons called for balancing supply-driven features by demand promotion: ICT and e-government projects in many countries had been unduly supply-driven. Project design featured awareness-building and skills development to promote demand but in reality ICT infrastructure and e-services supply were the primary focus. Assessment of project design: 2.1.3 The sheer complexity of project design, combined with its equally complex implementation arrangements, is notable. Project activities were largely foundational, whereas the original PDO and results indicators anticipated measurable outcomes more appropriate to a second stage operation: qualitative, behavioral and cost benefits of established systems, strong and measurable demand/usage, and with significant ICT capacity developed and consolidated (see 1.2.2). These are impacts which even in advanced economies can take up to a decade to reap measurable benefits. In practice, under the country conditions including a new client in a challenging political economy and economic slowdown from 2010-2013, and regardless of GOV’s strong commitment to an ICT-based economy, the complex foundation of infrastructure, systems and services as well as M&E capacity, could not be established fast enough to achieve and measure its development impact on the behavior of targeted beneficiaries by project end. The learning curve was steep and the configuration of involved institutions unwieldy. The Project did not get the opportunity to address second stage activities to enhance or measure, the impacts of the infrastructure on behavior. 2.1.4 Several issues affected design. First, bidding documents should have been ready prior to Board approval as part of the primary feasibility studies and would likely have led to better disbursement flow in the initial years. In retrospect, it is also clear that of the 92 procurement packages, some were not essential. Second, this type of project is not primarily about hardware and software – which can readily be purchased off the shelf - but about the eco-system for the human aspects. Project design lacked a substantial focus on the human behavioral change element which proved critical to its progress and to what would be measurable by end-project. Third, a higher portion of the project budget – perhaps as high as 30% - needed to be allocated to change management/initiatives. This would include, as with the procurement packages, getting 6 Enterprise Architecture (EA) itself or EA studies in place/done before Board approval, using Project Preparation Facility (PPF) or Trust Funds to ensure that the foundational elements of the Project could support subsequent procurement. Finally, the issue of ICT - and especially Internet - penetration was not adequately addressed resulting in e-services not evolving readily into end-to- end transactions. This project required something akin to a one-laptop-per-child approach or a subsidized laptop purchase program to accelerate the impact on citizen uptake. 2.1.5 Implementation structure: At preparation, multiple Project Implementation Units (PIU) were justified on the basis of project scope and the responsibilities of relevant government agencies. The project used a decentralized approach; each Subproject had its own PIU, procurement plan and Special (Designated) Account. The Bank’s conversation was quite different with each unit. The PAD argued that similar arrangements had been adopted by other Bank projects with multiple, decentralized subprojects or components. The real constraint was the policy and legal framework for subprojects financed by GOV’s Official Development Assistance (ODA) which does not permit joint PIUs across implementing/supervising agencies. 5 The PIUs operated as parallel “silos”; there was no built-in mechanism for fostering inter-PIU collaboration or synergy even though there were common components and similar objectives/outputs across Subprojects. Further, the PIUs were not constituted at appraisal and only three of the five were required to be up/running by effectiveness, which had implications for project ownership, understanding and readiness to launch. 2.1.6 The Subprojects were overloaded, each with multiple components/activities (see Annex 2) and IDA allocations ranging from US$10.0 m. to US$28.0 m., equivalent in practice to five separate projects. This was expected to implement smoothly in an untried, e-unready environment with ICT human resource deficits and varying levels of commitment to (or understanding of) the Project. As the Client was to bear the administrative costs, questions about duplication and inefficiency may have been discounted; from a Bank supervision perspective, similar questions were relevant. Selected cities covered the capital (to demonstrate e-government in the public and private sector), a commercially-developed city (Ho Chi Minh City) and one in a less-developed central region (Danang). The awareness that other donors were operating in the ICT “space” was reflected in plans for collaborative events on ICT/e-government with all key development partners in Vietnam. This was considered administratively manageable, would permit exchanges of information and avoid donor duplication of effort. In practice, it worked well. 2.1.7 Acknowledging the lack of clarity in individual institutional responsibilities – strong coordination being an essential feature of successful ICT operations – and complexity of the institutional and legal framework, the Bank Team sought coordination by linking the Project to the multi-agency National IT Steering Committee chaired by the Deputy Prime Minister, which would review project performance at semi-annual meetings. A Project Working Committee of PIU leadership was also to be established, meeting quarterly on project matters. 5 Each supervising agency must be responsible for its own “project” and form its own PIU for management. Bank staff in Vietnam has tried to rationalize this structure without success - it depends on the commitment of individual agencies. A modified approach could have worked where for example, a joint procurement team might have provided unified services for the three Subprojects based in Hanoi – MIC, GSO and Hanoi. 7 2.1.8 Participatory processes: The Bank project team worked with the involved public agencies and met with associations representing the private sector and with private actors in the ICT industry, incorporating their suggestions and priorities in project design. Private stakeholders suggested and endorsed the emphasis on streamlining government processes burdening their business operations. The PAD recommended that this dialogue – especially with SMEs - continue during project implementation and it did, but intermittently and depending on location. 2.1.9 Risk assessment: The analysis of risk was satisfactory but incomplete. Mitigation stressed training, awareness-raising and technical assistance. Risks also focused on concerns about commitment to and sustainability of ICT policies and systems. Procurement risk was correctly rated as High based on the readiness of bidding documents but the general lack of procurement expertise/human resources was barely mentioned. This was a procurement- dependent operation divided between five centers each with its own procurement plan and no familiarity whatsoever with the Bank’s procurement framework. The most acute risk evident after effectiveness was the limited capacity of implementing agencies (taking procurement processing into account) and lack of skilled IT professionals, i.e., e-readiness, alluded to at appraisal in general terms and rated Moderate. The 2010 Restructuring belatedly acknowledged human resource deficits and e-readiness as “new risks” for which mitigation measures were already activated, including more intensive/frequent Bank supervision. Risks associated with SME/private sector uptake, citizen ICT demand/literacy and design complexity were not mentioned. 2.1.10 Project financing: Financing arrangements were unusual. Six Special accounts were opened for the PCU and five PIUs. The Ministry of Finance (MOF) proposed that MPT, GSO and Danang receive their portion of the IDA Credit for their Subprojects as a grant. Hanoi and HCMC – which MOF argued had sufficient internal revenues to handle repayment - would receive their portions on a part-loan, part-grant basis. The two cities had requested that only those portions of their respective subprojects with cost-recovery or revenue-generating potential be financed through on-lending and the remainder through grants. The loan portion in each case was to be via advances in VN Dong from the state budget on modified IDA terms and with MOF bearing the foreign exchange risk, commitment and service charges. 2.2 Implementation Principal factors affecting project implementation: 2.2.1 Complexity and readiness to implement: Implementation conditions/capacity on the ground, especially for procurement and project/subproject management, was (except for GSO and Danang in key aspects) exceptionally weak, hampered by: (i) lack of experience with Bank projects/procedures and bureaucratic, multi-layered and cumbersome processing and approvals, especially by the People’s Committees; (ii) inadequate approaches at the PIU level magnified by their weak grasp of the Project’s opaque strategy and vision; and, (iii) in the first 2-3 years, frequent organizational and staff changes disrupting continuity and capacity-building, along with under-staffing, low salaries and difficulties in recruiting qualified/specialist personnel. By the 2009 Mid-term Review (MTR), the MIC and Hanoi PIUs still lacked essential qualified staff; (iv) since the MPT/PMU’s operating costs were not covered by the DCA until 2010, the unit lacked sufficient full-time staff to complete the Operational Manual or the website designed to aid inter- PIU communication; (v) implementation of government’s ICT/e-Government Strategy and its Administrative Reform Program was slow and poorly-coordinated, affecting the Project’s operational and policy context. The Project was integrated in the National IT Steering Committee early on but this mechanism for project orientation appears to have lost priority or involvement. 8 2.2.2 Procurement challenges: Procurement encountered specific constraints. First, contracting of international procurement specialists took inordinate time, delaying for up to two years in some cases the acquisition and installation of new infrastructure, systems and services not previously known/utilized in Vietnam. Second, inexperienced PIUs with no prior relationship to the Bank and negligible procurement capacity were required to execute 92 procurement “packages”, many requiring International Competitive Bidding (ICB). The 2010 Restructuring made marginal changes and dropped the HCMC Subproject, but essentially left this situation intact. Third, as noted in the Danang and GSO completion reports, Bank procurement no objections could be slow, and PIU procurement staff in general felt under-trained to use the Bank’s new approach introduced in 2010 for technical specifications in IT bidding documents; almost all bidding documents prepared required revision. Fourth, delays affected technology packages where rapid innovation left some outdated even before their purchase was concluded (see 2.2.3). Finally, GOV cost norms for specific packages could not compete with market conditions, further delaying official approvals. Given the scope of these problems, the Bank country management team authorized a senior procurement expert to provide hands-on advice to PIUs, with positive results, and with further follow-up support, the PIUs slowly mastered procurement processes. In general, PIUs with greater technical competence and strong, collaborative relationships with their People’s Committees (Danang, GSO) devised better technical solutions and more efficient procurement and contract implementation (see 2.4.2-2.4.3, and Section 5). 6 2.2.3 Rapid technological change: Related to 2.2.2, rapid technological change and fast evolving systems in the ICT marketplace from preparation through end-project affected progress and had important collateral effects. The marked reduction in ICT product cycles and increasing adoption of the enterprise architecture approach, broadband and Wide Area Networks (WAN), combined with the PIUs’ limited technical competence and institutional capacity, had the following effects: (i) Changing specifications delayed the development of detailed procurement plans with key packages identified in 2008, but bidding documents not ready for up to 18 months; (ii) PIU specialists needed time to update knowledge, digest consultants’ information and translate it into reasonable technical requirements. Some procurement packages were already superseded and could not be processed; (iii) Technical changes also prolonged the bid evaluation process for up to a year. Meanwhile, contracted equipment became unavailable requiring contract amendments and causing further delay. 2.2.4 Mid-term Review (MTR): The MTR was timely, well-staffed, accurately defined the primary obstacles affecting project execution and prepared the strategy and content of a proposed restructuring. Actions agreed as a result of MTR analysis included: (i) Level One restructuring incorporating elements discussed with authorities; (ii) maintenance by the PIUs of the same organizational structure and teams, backed by intensive training; (iii) closer involvement of and speedier approval of procurement packages by PIU leaderships; and, (iv) project-financed technical assistance to strengthen project management and supervise the execution of large procurement packages. Of particular importance, participating agencies (MIC and GSO) and 6 Well into project execution, when procurement issues had become chronic, the Bank team actively considered consolidation of the procurement function across PIUs as well as moving a Bank procurement staff member to work directly on the Client’s side on bidding and other documents. However, the associated difficulties were not easily overcome and the plan was set aside. The ICR suggests based on discussion with the current Bank team, that various solutions could have been devised to deal more decisively with this situation. 9 cities (Hanoi, Danang and HCMC - the latter dropping out after the Restructuring) were to streamline internal procurement appraisal and approval processes with clearer assignment of responsibility and accountability. Results of these decisions/efforts were inconclusive (see 2.2.5). 2.2.5 Restructuring 2010: The 2010 Level One Restructuring scaled the Project back while adding new activities but did not change its fundamental design and approach, or substantively address capacity constraints, perpetuating weak performance. The link between the revised PDO and Results Framework remained unclear, the PIUs still shied away from monitoring and measurement, and Bank supervision did not press the issue. Some activities were cancelled and the HCMC Subproject was dropped. The environmental category and applicable Safeguards policies remained as appraised. Expected benefits of this restructuring were increased numbers of beneficiaries of ICT development overall and better public access to e-services, especially in Danang to which an additional US$10.0 m. were reallocated consistent with its good performance. GSO also received an additional US$7.0 m. to scale up its Subproject. The PMU and PIUs’ limited project implementation capacity remained a risk but participating agencies and cities were to streamline internal procurement processes with clearer assignment of responsibility and accountability. As noted, and with certain exceptions, this restructuring did not improve progress. 2.2.6 High-level joint project review: Consultations in January 2012 between Bank senior management and the Vietnamese Deputy Prime Minister (DPM) galvanized progress in all PIUs and were a decisive factor in ensuring completion of the Project. Notably, some 10 Bank- supported projects in Vietnam were performing poorly at that time and were on the Bank’s watch list; the ICT Development Project was not an outlier. Responding to the Bank’s insistence on action, the DPM issued direct instructions to participating agencies to accelerate execution. Intensified Bank supervision, including a technical procurement mission provided direct support to critical procurement packages linked to achieving the PDO. Detailed action plans were discussed and agreed with the PCU (MIC), PIUs, the Ministry of Planning and Investment (MPI), the State Bank of Vietnam (SBV) and the Ministry of Finance (MOF) on procurement, financial management, and results monitoring. Substantial progress resulted by January 2013 with most outstanding procurement packages signed. Commitments increased to 85% of the total IDA credit. 2.2.7 Proactivity of Bank supervision: Changed Bank task management, combined with the actions described above (2.2.6) reversed the project trajectory and built renewed trust. Further restructuring in mid-2013 extended the closing date and adjusted the Results Framework for clarity/relevance. The project disbursed rapidly in 2013 because a large number of contracts initiated in 2012 were completed. The Bank Task Team provided intensive technical implementation support with additional IT experts, helping to shorten the technical review of bidding documents and bid evaluation reports to the Bank’s service standard of an average of two weeks. M&E was prioritized: PIUs designated individuals responsible for specific segments of the process and monthly updates on results were conducted via video-conferencing. The risk of project cancellation, continuing relevance of the PDO, and importance of the Project to the Bank/GOV relationship and dialogue, pressured the Bank and country counterparts to complete outstanding procurement/training, and ensure that equipment/systems were operational by closing. The Bank portfolio review made senior management in supervising agencies aware of pending issues and take greater control of procurement. But above all, counterpart staff was able to “master” the Bank’s ICT procurement guidelines especially the new regional guidance on the shift from a specification-based to performance-based approach. 2.2.8 Project at Risk Status: The MTR downgraded the project ratings to MU and the Project fell into problem status. Performance remained sluggish until 2012 but, consciously erring on the conservative side given the Project’s generally troubled trajectory, the DO rating was maintained 10 by the Bank team at MU through closing. The IP rating was upgraded to MS due to rapid and conclusive progress on remaining procurement packages in 2013. See Annex 2 matrix of procurement packages completed. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 2.3.1 Design: The PAD called for a Management Information System (MIS), complemented by semi-annual progress reports and impact studies prepared by the PIUs and PCU. This was consistent with the PAD’s assertion (Annex 1) that the challenge facing government was not only how to achieve its ambitious targets but to strengthen the results-orientation of several ongoing programs through the “establishment of realistic indicators and benchmarks for rigorous M&E”. Monitoring and evaluation (M&E) was to be the responsibility of individual PIUs at their subproject level with the MIC PCU consolidating results sent by the PIUs, identifying lessons and disseminating knowledge to stakeholders - including the Bank and government. 2.3.2 Evolution of the PDO and Results Framework: The ICR Data Sheet illustrates the problematic Results Framework. The original PDO was broad, complex and in key aspects unmeasurable within the life of the Project. The causal link between the Project Outcome Indicators and PDO internal elements was unclear, and confused outputs with outcomes. Presentation of the PDO was inconsistent within the PAD, including the Results Framework (RF) and Arrangements for Results Monitoring (ARM), and between the PAD and Development Credit Agreement (DCA), creating difficulties for the ICR. Overall design of the RF was exceptionally weak, with excessive numbers of indicators (especially Intermediate Outcome and the separate slate of outputs tied to the components – see Annex 2) and many were not SMART (specific, measurable, achievable, relevant and time-bound). The ICR adheres to the initial, two- part statement of PDO which also accords with the DCA and as shown earlier in Sections 1.2 and 1.3, aligns the results indicators accordingly. 2.3.3 The 2010 Level One Restructuring narrowed PDO scope to focus more tightly on IDA activities and added several Core Indicators concerning processing time for public services and user perceptions of e-services. Overall measurability improved to some extent, but the number of indicators remained unusually large and their ability to capture the PDO was still problematic. Retention of the private sector objectives was not matched by appropriate tools to deliver the impacts sought. Revisions were overly-focused on input indicators measuring implementation rather than results. No plans were made to conduct an impact evaluation or related study - surprising given that M&E standards in the Bank were much higher by 2009 - virtually no monitoring activity had yet occurred, and this was the logical opportunity to formalize M&E. 2.3.4 The Level Two restructuring in June 2013 further adjusted the RF, while leaving the PDO intact. The new Project Task Team in 2012 mapped key project activities and procurement packages against expected outcomes reflected in the internal elements of the PDO, and found that large numbers of important activities directly affecting PDO achievement were still in the early stages of procurement. The decision to further adjust the RF to adequately measure achievements of the PDO improved measurability of indicators per se and linkage to the PDO, but the overall number of indicators - including many with multiple internal targets - increased (see Data Sheet). 2.3.5 M&E Implementation: Overhaul of both the PDO and indicators in 2010 did not result in a sharper focus on monitoring, and evaluation lacked priority. Continued poor definition/clarity 11 of the RF, limited capacity and focus in the PIUs and continuing inability to see the Project in integrated terms, plus the PCU’s general lack of leadership deterred data collection on project results. The PIUs were monitoring progress on the procurement packages. 7 In a major push to have the PIUs focus on PDO achievements, each assigned an M&E focal point for tracking progress and key procurement packages were accelerated, but the overall quality of data reporting was low and collected data was not used for planning or decision-making. Data collection and organization improved in the final 18 months and evaluation/studies made headway, including: an independent evaluation of the Danang Subproject (Leading Associates/KDI Asia, 2013) paving the way for rigorous M&E of future, similar projects; a well-conducted survey contracted by MIC PCU (An Vien Media, 2013) to gauge the effectiveness of the project communications campaign (Annex 5); and, a Project Performance Assessment Report (KDiAsia), intended to provide MIC with internal guidance for future operations. 8 MIC/PCU, Danang and GSO prepared Completion Reports (executive summaries in Annex 7). 2.3.6 Utilization: The ICR acknowledges some internal Bank concerns that the adjustments to the Results Framework in mid-2013 seemed too late to be used to measure end-project outcomes, but opted to use that final version anyway, for the following reasons. First, initial analysis prompting the decision to adjust the RF occurred in March 2012 with the expectation that changes would be effected within a reasonable period. However, the multi-layered, multi-agency sign-off process in GOV delayed final Bank approval until June 2013, six months before the revised closing date and 15 months after the initial proposal to restructure. Second, adjustments in 2013 were a serious effort to ensure that the indicators reflected actual project achievements. Indicators were further refined and brought down to project level by specifying units of measurement, linking targets to the institutions where results were expected to occur, eliminating intermediate indicators considered irrelevant, and adding two new Results Indicators to more accurately cover expected PDO outcomes. Such changes were also required by the cancellation of some procurement packages and amendments in the project description. Third, the project archive shows that senior Bank management tacitly assumed that the Level Two adjusted RF would be the version used to measure achievement of the PDO. 9 2.4 Safeguard and Fiduciary Compliance 2.4.1 Safeguards: Due to its nature, the project was an environment Category C, and did not trigger any safeguard policies. The category did not change with restructuring. 2.4.2 Procurement: Procurement execution involved “learning by doing”. Procurement risk was rated High at appraisal because the PIUs had not yet been established nor their procurement 7 Evidence suggests the PIUs – GSO and Danang excepted - always saw the Project as an aggregation of procurement packages since their focus was almost wholly on getting those prepared, approved and executed. Most PIUs lacked any concept of the “Theory of Change” although at end-project they were all able to narrow down the Project’s key impact on behavior – the transition from manual to electronic administration and services provision. 8 See: Impact Assessment Study of Computerized Services Delivery Projects from India and Chile, Bhatnagar, Tominaga, Madon and Bhatia, World Bank Staff Working Paper, #42147 (November 2007).The study demonstrates how to estimate direct and indirect benefits to clients with reasonable accuracy using a framework which includes surveys of clients (citizen and business users) and other stakeholders. While the study shows the potential of ICT to help reduce costs of accessing services, improve convenience and accountability and promote good governance, it also advises looking beyond the “hype” of e-government and that the principles of good project design and management remain important to results. 9 Email from U. Zachau (EAPVP) to V. Kwakwa (EACVF) of January 11, 2013 refers to key remaining issues in project implementation for the 6-month extension period including the need for “….modification of the indicators to measure more accurately the results contributing to the achievement of the project development objectives….”. 12 staff identified. Many procurement packages, including specialized consultant services were unique and challenging, the cycle of no objections and PIU approvals was inordinately protracted in many cases, and some PIUs were reluctant to communicate their need for Bank guidance. Difficulties arose - and persisted over time – in getting agencies’ clearance/approval of project procurement documents due to their insistence on applying the National Procurement Law despite provisions in the DCA - and in the National Procurement Law itself - giving precedence to Bank Guidelines. As discussed, procurement performance was variable between PIUs and greatly affected the Project’s ability to advance (Section 2.2). Most IT projects face similar procurement challenges. The Bank may need to revise its procurement documents for IT projects, moving beyond an input-focused approach to processes more appropriate to a fast-moving sector. 2.4.3 Within this context, the Bank procurement team (based in Hanoi) conducted regular supervision, training, workshops and direct mentoring in procurement planning, cost estimation, terms of reference, technical specifications, RFPs and bidding procedures. Procurement ratings fluctuated between MS and MU over the project period. Complaints associated with procurement packages in MIC and Danang in 2012 resulted in explicit warnings to all PIUs and additional resources to focus attention on bidding processes and transparency. The MU ratings reflected slow procurement progress and weak management, not malfeasance. The PCU/PIUs grasped the importance of (and strove to develop) sound quality control of bidding/other processes and to comply with Bank standards, slowly overcoming deficiencies with time and mastering processes. 2.4.4 Financial Management (FM) and audit: As with procurement, FM required intensive attention/supervision. The FM performance rating fluctuated between MS and MU over the life of the Credit and financial risk was rated mostly Substantial. The main reason was variable capacity between PIUs/PCU. At appraisal and as with procurement, there was no institutional base or identified staff for project FM, and no computerized system (MIS) in place to track project transactions and conduct financial reporting. Institutional arrangements with multiple units and Special Accounts affected FM efficiency. Internal controls, including contract management, eligibility of expenditures, defined FM responsibilities, reporting structures and accounting software – the latter experiencing serious problems of installation, activation, and systemic errors depending on the PIU – were slow to improve. Capacity-building was recommended and training provided. The rating fell to MU in 2012 due to a series of outstanding issues - ineligible expenditures, delayed financial audits and weak contract management - but was restored to MS by the final supervision due to all PIUs showing improved performance in specific aspects. 2.4.5 Audit: Eight audit reports were required and by ICR, seven were received with one pending. Three audits were late and two had issues of accountability and internal controls. Auditors’ opinions were qualified on occasion and audit quality was uneven. Delayed delivery of the 2011 audit and protracted delay in reimbursing the Bank for some ineligible expenses implied the possible suspension of disbursements under BP 10.02, but issues were resolved to the Bank’s satisfaction by closing. 2.4.6 Project costs: Total project cost was US$77.08 m, 72.1% of the US$106.96 m. estimated at appraisal. 10 Cancellation of IDA resources totaled US$13.52 m. over the project 10 Total disbursement of US$74.12 m, cancellation of US$13.52 m and undisbursed balance of US$6.23 m, as shown in Bank system, June 26, 2014. Recipient contribution is a best estimate as at same date. See Annex 1. 13 period. The Recipient’s contribution was an estimated US$2.96 m, 22.36% of the appraisal estimate. Costs per Subproject varied significantly from appraisal estimates (see Annex 1). Combined supervision costs for FY12 and FY13 alone were US$384,000, well above regional norms due to the marked intensification of Bank supervision in that period. Due to the tendency for prices to fall, and increasing efficiency in procurement management over time, project costs were reduced by a total US$15.4 m. over the course of the Project. More effective and cheaper solutions became available such as the scanning facility in GSO and online visa application system in MOFA. The Danang PIU Completion Report cites combined savings on several of its packages exceeding US$6.0 million. 11 The PCU and all PIUs used savings to purchase additional systems/equipment. 2.5 Post-completion Operation/Next Phase 2.5.1 The Bank and counterpart project teams mapped out a plan for the immediate post- completion operation and maintenance of project infrastructure and systems and their medium- term sustainability (see Section 4). As discussed (2.3), no formal O&M system was established by the Project although GSO by the nature of its data and research functions is well-positioned and motivated to maintain project achievements, as is Danang City which has become a national e-services demonstrator. 2.5.2 An official delegation from Vietnam visited the Bank in December 2013 to present the results of the ICT Development Project and to outline a new ICT operation which might include scaling up the especially successful Danang experience nation-wide, further increasing the scope and efficiency of the GSO, and in the case of Hanoi, replicating and increasing e-services across different levels of government. Given the foundational nature of the Project, continued development of IT professionals and adequate budgets are essential. Annex 5 summarizes recommendations of the Danang study which point the way to post-completion operation and expansion of this/similar projects. Regardless, the ICR suggests, based on reviewers’ comments, that any new ICT intervention be results-based or a Development Policy Credit rather than an Investment Lending (IL) operation. A programmatic approach for such projects, with a second operation contingent upon performance under a first, may be advisable. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Rated: Moderately Satisfactory 3.1.1 Relevance of PDO: Continuing relevance of the (restructured) PDO is rated Satisfactory, capturing key development themes in the current Country Partnership Strategy (Report #65200-VN of November 7, 2011) and GOV’s current development priorities. The CPS identifies issues affecting the business environment: weak infrastructure, high costs of interacting with government; and, lack of adequate information on policy, all of which resonate directly or indirectly with the PDO to “foster private sector development by facilitating the increased adoption of ICT”. The CPS recommends, inter alia, strengthening Vietnam’s technology and innovation system, and supporting the country’s transition to a successful mid-income country by 11 Danang PIU US$6.0 m, MIC PIU US$3.7 m, MIC PCU US$0.4 m, GSO US$3.7 m and Hanoi US$1.6 m 14 promoting innovation-led and higher-value-added production and services. Strengthening governance is an important cross-cutting theme and links directly to the PDO: “support MIC’s institutional development and technical and regulatory capacity for ICT oversight and to increase efficiency and transparency of public service delivery in MIC, GSO, MOFA, Hanoi and Danang”. 3.1.2 Relevance of design and implementation: The relevance of project design and implementation showed significant shortcomings, and is rated Moderately Unsatisfactory. Prima facie, the technical design was acceptable and responded to defined needs, but the Project’s ambitious vision for the project period was unreasonable on capacity and readiness grounds. The risk assessment was incomplete, under-estimating human resource factors, and overlooking issues such as the demand side, citizens’ IT literacy/uptake and the implications of overall complexity. Mitigation measures were limited/inadequate. The relevance of PDO and Results Framework design was especially weak; the former improved in 2010 but the latter not until the final year. Implementation of M&E was unsatisfactory until the final stages. Few specific activities were tailored to private sector needs or linked to the PDO’s private sector goals. Most of the Sub-projects (components) were implementation inputs and outputs linked causally to improved government services. While the private sector would, like regular citizens, be likely to benefit from many of those services, the link to “private sector development” was not well- justified or measurable. The institutional arrangements fostered inefficiency, duplication and delay throughout the project cycle. They would today be subjected to tougher internal scrutiny and demand joint institutional arrangements and shared functions. The PAD discussed the need to sequence business process change and awareness-building ahead of equipment purchase and application development but in practice, project implementation was supply-driven and the human factor was under-emphasized. The demand side prominent in current Bank policies/methodologies across sectors lacked priority. 3.2 Achievement of Project Development Objectives 3.2.1 Revision of the PDO via a Level One Restructuring entails that the ICR conduct a split assessment of outcome using the pre- and post-restructuring PDO and indicators, weighted based on respective levels of disbursement in the two stages. The following presents project results against the original PDO and indicators up to the date of Board approval of Level One changes (January 29, 2010). Results based on the revised PDO and RF up to closing at end-December 2013 are shown separately using the indicators as restructured in June 2013. Pre-restructuring period (from Effectiveness to January, 2010) 3.2.2 Profound delays in project execution and disbursement under the first stage of the project resulted in few PDO-linked achievements in this period. To the extent that any measurement was conducted, it was of basic progress in the contracting and completion of planned procurement packages which for the PIUs – both at that time and virtually throughout – were synonymous with the Project. Original PDO Part A: Rated Unsatisfactory 3.2.3 The internal elements or themes of the original PDO Part A were four-fold: “to accelerate the use of ICT and support institutional development; to increase the efficiency and effectiveness of public administration; increase the transparency and availability of information; and, improve public services delivery to stakeholders”. With two notable exceptions, there is little to show through end-2009, evidence of the acute problems affecting implementation. All four elements entailed a critical mass of investments and appropriate evaluation including user surveys and case studies to measure behavior change through gains in “efficiency”, “effectiveness” 15 and “transparency”, including several at the level of national indices and probably unmeasurable even under higher levels of achievement at the time. 3.2.4 The following situation prevailed at end-2009: • MIC issued common technical standards for the Inter-operability Framework in 2008, and guidelines for the development of the E-Government Enterprise Architecture (EA) in 2009. These were important milestones for Vietnam’s overall ICT development. • Danang City put a city-level EA in place to guide its e-government development. • GSO acquired and developed its Form Scanning system crucial for faster processing of the 2009 Population Census. • Otherwise, negligible achievement was shown by most PIUs (except Danang and GSO) in acquiring the infrastructure, equipment, systems and services to move beyond the policy and standards stage. • By end-2009 citizen usage remained incipient, as did the “increased transparency and availability of information and improved public services delivery to stakeholders”. Original PDO Part B: Rated Unsatisfactory “Foster private sector development in particular by facilitating the increased adoption of ICT by small and medium enterprises (SME)” 3.2.5 There is in the pre-Restructuring period, some limited evidence of SME adoption of project-financed ICT but no hard evidence that this “fostered private sector development”, a complex linkage difficult to measure in the circumstances. The Bank MTR team reported that SME-related activities had received modest attention in the period. 3.2.6 Addressing individual indicators: • Anticipated growth of 30% per year in the domestic IT service sector as a result of the project fostering private sector development was unrealistic and lacked a baseline on which to assess incremental gains. • Facilitation of “increased adoption of ICT” was limited to Danang having established its ICT Transaction Center (TC). • 15% of SMEs in Hanoi, Danang and HCMC were to be using ICT for business operations by 2010. The baseline is identical for all three cities and is assumed to be national, not city-specific: Danang, measured via canvassing of SME customers using the new TC, registered 18% usage for business purposes by early 2010, but attribution to the Project is unclear, i.e., whether the increase is based purely on impact of the TC on awareness. Post-Restructuring period (February 2010 to end-December 2013): 3.2.7 Delays continued to plague the post-Restructuring period and for many of the same reasons, but Danang City and GSO continued to lead the way and overall, progress picked up. By closing, all procurement packages not cancelled were fully completed and operational. Evidence is mostly output-based with one exception: Danang PIU financed a study which paves the way for future M&E of ICT development projects and reports findings and lessons directly attributable to the Danang Subproject.12 Some results are included below, with further details in Section 3.6 and Annex 5. 12 See DNG14: Assessing the Status of Danang e-Government Implementation and Building its e-Government Model based on the Assessment. Leading Associates and KDi Asia, 2013. 16 Revised PDO Part A: Rated Moderately Satisfactory “To support MIC’s institutional development and technical and regulatory capacity for ICT sector oversight and to increase the efficiency and transparency of public sector service delivery in MIC, GSO, MOFA, Hanoi and Danang”: 3.2.8 All PIUs established the foundation for improved e-government, formulating frameworks and standards, upgrading key e-government infrastructure, and training human resources at managerial/other levels, both at HQ and in decentralized government units. A fundamental transition occurred from manual to computerized administration and services. Internal procedures and public service systems were improved or developed using ICT and are fully operational including portals and websites. These are likely to have improved the efficiency and transparency of public services delivery, democratic access to information, and government accountability, although the ICR cannot definitively prove this even from the Danang study. Danang and GSO are good cases illustrating how an ICT project can support local e-government and a central agency respectively, to become service- and citizen-oriented facilitators/providers. The following assessment disaggregates the PDO by its main themes. Theme 1: To support MIC’s institutional development and technical/regulatory capacity for ICT sector oversight: • MIC’s institutional development was supported through the formulation and application of four standards: Enterprise Architecture, the Interoperability Standard, the e- Government Framework and Security Standard. • MIC’s regulatory capacity was improved through laws, decrees and circulars promulgated/approved: Postal and Radio Frequency Laws; the Information Security Law – of crucial importance in an expanding e-services environment - and MIC decrees supporting the application of IT and Information Security. • Technical capacity improved through numerous infrastructure upgrades: LAN networks in MIC headquarters and 12 DICs; smart building communications systems connecting MIC to regional offices in Danang and HCMC, MOFA and the Hanoi Telecom Authority; video-conferencing, IP phone and building security systems operational. • At the district level (DIC), upgrades included: PCs, laptops and printers, video- conferencing facilities, email and LAN/WAN integrated connectivity. • Training of 500 CIOs/IOs, 220 state-owned enterprises and 70 SMEs supported the operations and sustainability of these upgrades (policy formulation, capacity-building and implementation guidelines for IT and telecom sectors). Theme 2: Increase the efficiency and transparency of public sector service delivery (MIC, GSO, MOFA, Hanoi and Danang): • The primary change in behavior was the transition from manual to electronic/digital processing of governments’ key business processes and delivery of public services. • Reduced transaction times demonstrate efficiency gains: MIC’s Radio Spectrum processing time was halved from 20 to 10 days; registration for imported publications 17 from 30 days to 15; online visa application processing averaged 2.5 days; in Hanoi, average transaction time fell 50% for selected e-government services via the City Portal. • GSO’s Population Census data collection/processing time was reduced by 42% with an estimated cost reduction of US$500,000 as a direct result of project-supported ICT upgrades; Forms Scanning infrastructure and other upgrades facilitating the 2011 Agriculture and Rural Census reduced processing time by 50% (two years to one); and, data collection/processing for CPI surveys fell by one-third (15 to 10 days). • Global experience validates the tendency for online services to promote transparency. The Project established, inter alia, online drivers’ licensing, business registration, water quality monitoring, construction permits and public transport management. • Improved connectivity and information flow G2B, G2G and G2C, as well as local facilities such as transaction and data centers, information kiosks, online booths, e- Learning portals and public WiFi also likely promoted efficiency and transparency. 13 Revised PDO Part B: Rated Moderately Unsatisfactory Foster private sector development by increasing access to business-related government services and raising awareness and usage of ICT in Hanoi and Danang. • Interviews and anecdotal evidence suggest that the “viral” nature of ICT skills acquisition, the breadth of information now available to business and speed of access to it, and efforts by the Project to promote awareness, are positive signs for downstream gains in private sector development beyond the Project. • Substantial numbers of SMEs took ICT training courses, visited/utilized the new Transaction and Data Centers, and accessed online e-services, e-learning courses, portals, and SME-tailored training modules. • Localized surveys in Danang showed that 33% of SMEs were using ICT and 18% using it for business purposes. Caveats noted above (3.2.6) apply. • SME’s awareness of the availability of project-financed, business-appropriate e-services and how to use them had increased. • Several business-related e-services, e.g., online Business Registration and Licensing, and the Investment Project Management System were launched in Hanoi and Danang, markedly reducing transaction times compared to legacy, paper-based transactions. • However, despite these gains, there is no hard evidence that private sector development was fostered only that access, awareness and usage by SMEs had improved. This indicator was felt to require more formal evaluation to reveal specific evidence of “development” and that the behavior changes noted did not in themselves, equate to that development. 3.3 Efficiency 3.3.1 Assessing the efficiency of the Project in a rigorous manner would require extensive data, including detailed financial and operating information from government agencies as well as surveys carried out among beneficiaries both before and after the interventions. Given that this data was not available by project closing - and experience shows that a period of maturation is 13 The Danang study (Leading Associates/KDiAsia, 2013) found that – at least at project closing - a majority of potential and actual citizen e-service users interviewed expressed preference for a human face at the “counter” but this is not interpreted as dissatisfaction with online services per se. E-services were still tending to be used for information- seeking not end-to-end transactions. Many factors constrain demand including computer literacy and access to a computer per se, which received insufficient attention from or acknowledgment by, the Project. 18 normally needed for ICT investments before such data is meaningful - certain assumptions can be made about the efficiency of the project based on results as well as what is likely to happen based on experiences in other countries. See Annex 3 including sources. 3.3.2 Project interventions affected efficiency for three groups: government, businesses and citizens. For government, transitioning from manual to computerized processes results in lower costs and shorter processing times. Project-financed training of government officials taught them how to implement IT systems to achieve efficiency gains. Further, sensitizing high level officials is likely to have reduced resistance to restructuring business processes. Project-fostered coordination among government agencies will have reduced duplicate IT systems resulting in improved efficiency. 3.3.3 GSO: Data collection and processing for the 2009 Census was reduced to 14 months from 24 months (42% reduction) as a direct result of project ICT improvements. The estimated cost savings were US$500,000 based on the 450 data entry staff used for the Census, assuming they received the average monthly salary for government workers. Forms scanning hardware used for the 2011 Agriculture and Rural Census generated a 50% improvement in overall processing time, reduced from two years to one. Data collection and processing time for Consumer Price Index surveys was reduced by one third from 15 to 10 days using newly-introduced IT processes. The GSO subproject also reduced duplication by harmonizing and integrating databases within a standard system architecture. The provision of additional/free statistics online would also have improved decision-making by both government and the private sector. 3.3.4 Small and medium enterprises: Interventions aimed at Small and Medium Enterprises (SMEs) included training and online services. These are likely to have positively impacted SMEs in Vietnam. World Bank research suggests that online business registration systems (among the applications developed by the Project) reduce complexity, time and costs leading to a rise in business registrations and employment (as well as generating income for government). Similarly, facilitating construction permits (another project application) through online applications saves businesses time and money. 3.3.5 Danang City e-services: The delivery of online public services benefits citizens by reducing the cost and time of travelling to and waiting at government offices. Examples cited in Annex 3 include Sri Lanka and Europe. Similarly, 45% of respondents to the Danang survey expected project-financed e-services to save time, 35% foresaw greater convenience and 23% expected to save costs. The city-wide free Wi-Fi scheme in Danang saves citizens time and money. Businesses also benefit: customers could find businesses easier using Internet-based location services, and businesses could exploit social media/other applications to create innovative relationships with customers. Estimating the cost savings to citizens from free Wi-Fi means calculating the cost of Internet access payments no longer needed. Assuming the same proportion of Internet users in Danang as the nationwide average and that one third would use the free citywide Wi-Fi service, the aggregate consumer benefit is over US$350,000 per month. 3.3.6 Overall project implementation efficiency was also notable. For example, the total estimated value of the Danang subproject at appraisal was US$27.3 million. After five years of implementation, Danang PIU saved a total of US$6.1 million through competitive procurement practices and good management, and acquired additional equipment with the savings. All other PIUs also registered substantial savings although not on the same scale as Danang. Aggregate cost savings were US$15.4 million. See also 2.4.6 above and footnote 10. 19 3.3.7 Although detailed information necessary for carrying out a comprehensive efficiency analysis is lacking, the anecdotal data suggests that project interventions are already having an impact on reducing costs for government and saving time for SMEs and citizens. These benefits can be expected to grow as e-services evolve in scope and functionality and more Vietnamese use the Internet. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory 3.4.1 Overall PDO rating: Consistent with the ICR Guidelines Appendix B, separate outcome ratings for the original and restructured projects respectively were weighted in proportion to the share of disbursements in the periods pre- and post-Restructuring. PDO Rating - Original and Restructured Project Item Original PDO Revised PDO Overall (a) Rating Unsatisfactory Moderately Moderately Unsatisfactory Unsatisfactory (b) Rating value 2 3 - (c) Weight (% of total available 10% 90% 100% financing disbursed, original and restructured project) 14 (d) Weighted value (c x b, 0.2 2.7 2.9 rounded) (e) Final Rating - - Moderately Unsatisfactory 3.4.2 Overall project outcome rating: This rating of Moderately Unsatisfactory takes into account the following: • Relevance: Assessed against current Bank strategies, PDO relevance remains Satisfactory. However, the ICR asserts that a technically and operationally demanding project design was largely imposed on an inexperienced client. Capacity and readiness to implement were over-estimated at appraisal. Design and implementation relevance are therefore rated Moderately Unsatisfactory. Overall relevance is rated Moderately Satisfactory. • Efficacy: Achievement under the original PDO is rated Unsatisfactory because with the exceptions noted, progress as measured by level of outputs (completed procurement packages) and disbursements was still negligible by end-2009 and thus most expected outcomes – changes in behavior – lacked the conditions for achievement. Most of the latter came well after the 2010 Restructuring and some of the most critical not until 2013. The post 2010 Restructuring phase is rated Moderately Unsatisfactory, reflecting a conservative approach which acknowledges the Project’s continued difficulties, last minute execution of key procurement packages and lack of time to conduct user surveys, although the vast majority of planned systems, equipment and training were implemented, with substantial cost savings, and with early/preliminary indications of reduced transaction costs and greater efficiency. However, looked at overall, and not just through the lense of the good performers (Danang and GSO), there is a significant risk given likely medium and longer-term challenges, that outcome may not be sustained (see Section 4). 14 The ICR takes total IDA resources disbursed as at July 16, 2014 (US$74.07 m), and for the pre-Restructuring period, total disbursements by January 28, 2010 (US$7.35 m) as a percentage of that total, with the balance attributed to the second, post-Restructuring stage. This results in relative percentages of 10% and 90% respectively. 20 • Efficiency: Based on evidence of efficiency gains in time and cost already reported and likely impacts based on global experiences for public e-services, balanced by the acknowledgment that the project currently has limited hard evidence of citizen/SME gains, and the fact that the efficiency discussion is largely couched in terms of likelihoods, project efficiency is rated conservatively as Moderately Unsatisfactory. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 3.5.1 The project had no overt poverty objectives, and did not address gender either in design or during implementation perhaps due to a perception that they were not immediately relevant goals for the type of project and its stage of development. Neither did the project have explicit social objectives although it can be inferred that more efficient provision of government services and the wide range of e-services already developed in the Project’s lifetime, have solid potential for positive social impacts, e.g., water quality monitoring introduced in Danang; more rapid and transparent business registration; e-learning portals and TV programs delivering courses to enhance citizens’ skills; and, improved management of public transport. Potential social benefits are likely from better planning and policy-making using GSO’s improved population/other data and modernized systems/procedures. (b) Institutional Change/Strengthening 3.5.2 The project has already had significant impact on the administrative and operational efficiency of the participating institutions, most notably (and measurably) the General Statistical Office (GSO). Key government services are more accessible in less time to current and potential users. Most key administrative and operational functions are now electronic/digital rather than manual, representing the most profound of the behavioral changes prompted by the Project. Most staff from the PIUs, now possessing specific skills developed over the course of the Project, have already returned to their parent agencies. Danang representatives refer to their daily activities as being more “citizen-centric” and that coordination and inter-connection among departments and agencies is more streamlined and easier. The Project laid the foundation for changing the science of service delivery and inclusive innovation with ICT. Other new innovation, public procurement and education projects in the Vietnam portfolio will benefit from this achievement. (c) Other Unintended Outcomes and Impacts (positive or negative) 3.5.3 Examples of unanticipated developments include: (i) substantial, unexpected aggregate cost savings of US$15.4 m resulting from competitive and over time, more efficiently-managed procurement; and (ii) collaboration with other donors conducting similar/related projects (as envisaged in the PAD) went further than expected. For example, the close consultation/coordination between GSO and the United Nations Population Fund (UNFPA) within the design and implementation of package GSO 6.1(b) provided more efficient data entry and processing technology for the 2009 National Population Census. UNFPA also helped design GSO’s questionnaires appropriate for mechanized data entry and optical recognition, and trained interviewers and enumerators. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 3.6.1 The Danang Subproject financed the project’s only major evaluation study, but the MIC PCU also financed an important survey of citizens’ awareness of the ICT Project and their opinions concerning the benefits of project-financed e-services. Relevant conclusions from 21 Danang are summarized below, with details on both reports in Annex 5. The Danang study used the e-GAF (e-Government Assessment Framework) which looks at ICT Outcomes, Program Management and Governance, using qualitative, interview-based information in this case from 642 citizens, 75 businesses and 38 public agencies, including transport, health, science and technology, Peoples’ Committees and DICs. The findings of three other methodologies discussed in the Danang study are summarized briefly in Annex 5 but their research ranged more widely and only a small portion of results can be attributed directly to the Project. An important objective of the Danang study was to illustrate methodologies available for future ICT M&E. • Project-financed Danang ICT Transaction Center, Data Center, Public WiFi and Metropolitan Area Network (MAN) are providing good quality services to the public and government agencies. • E-services being quite new, their current role is focused more on information than transactions. Citizens’ IT skills/access remain limited and information campaigns about e-service availability/usage were still ongoing at the time of this study. • With more services migrating online and demand increasing, human resource capacity and budgets must also be scaled up with attention focused on managing information security and measuring ICT performance. • Danang’s ICT strategy is based on consultative, inclusive processes involving stakeholders and awareness of ICT is high among senior city managers interviewed. • Due to the Project, Danang City’s e-government was rated top of the Vietnam ICT Index for five years (2009-2013) and its ICT development is entering a maturing phase. 4. Assessment of Risk to Development Outcome Rating: Significant 4.1 Final supervision missions and video-conferences associated with ICR preparation obtained opinions from the PIUs and MIC/PCU on factors likely to foster the maintenance of installed systems. In the country context, waning interest/investment in the ICT sector is not seen as a risk either at the central or provincial levels, while recent legal directives, resolutions and decisions underscore Vietnam’s commitment to ICT in industrialization, modernization and institutional development. Nevertheless, significant risks include: technical, since the project financed a large amount of innovative technology and systems; financial as indicated by the need for adequate budget to fund operation, maintenance and replacement costs as well as human resources development; and social, the latter equated with citizens’ IT literacy and building sustained demand for the services and facilities implemented. The ICR believes it probable that the achievements of strong performers like Danang and GSO will be sustained, but uncertainty is associated with MIC and Hanoi, the assessed project benefits to date are couched in terms of likelihoods based mostly on similar external experiences, and overall sustainability of the Project must be questioned given that even its restructured stage is rated Moderately Unsatisfactory and several large procurement packages were not completed until the final months. 15 Final supervision missions, and video-conferences associated with ICR preparation discussed with the PIUs and MIC/PCU the factors likely to ensure that project development outcomes (or expected outcomes) would be maintained/realized. 15 The project team opted for a conservative approach to rating the post-Restructuring phase, deciding on MU to acknowledge its continuing difficult trajectory, last minute execution of key procurement packages and lack of time to conduct user surveys, although the vast majority of planned systems, equipment and training were implemented and operational, with substantial cost savings, and with early/preliminary indications of reduced transaction costs and greater efficiency. 22 4.2 Technical: GSO intends to increase public access to its statistical databases, standardize additional data sets and install them in its statistical hub for use across departments, open further databases to public access, maximize the data mining capacity of new systems, and pilot automatic on-demand report generation and provision. Hanoi and Danang have specific plans to expand e-services availability while increasing ICT uptake/usage. In most cases where relevant, suppliers delivered O&M training and both national and local authorities are well-informed on the budget needed to maintain connectivity, especially for Internet. The issue of ICT security - flagged in the Danang study - has lagged and needs attention, while rapid technological change, obsolescence (including of the Enterprise Architecture) and development of a national ICT innovation sector are issues of broad national interest and relevant to sustainability. 4.3 Financial and institutional: Project-financed infrastructure, e-services and public facilities will remain operational. Adequate budget is available in most cases to maintain project- financed equipment, infrastructure and e-services, except for MIC/PIU which is still negotiating with related departments in MIC itself, MOFA and MPT for the budget needed for the rest of 2014. MIC’s task is more complex, covering the public sector, citizens and business. It has requested budget for M&E of e-services at the provincial level. Most project-trained staff have returned to their respective agencies with expected institutional benefits, and public e-facilities are open for business and adequately-staffed. PIU teams are aware that extending e-government facilities to other public services entails risks in terms of the availability of specialized/trained human resources, adequate budgets and demand. GSO is recommending that the Statistical Law be updated to create the mechanism for fee-based information services. Dissemination of project results and lessons to stakeholders and promoting demand is an acknowledged priority. 4.4 Social: Recognizing that citizens’ ICT literacy is comparatively low, Danang launched new Citizens Services Centers (one-stop facilities) to build familiarity with ICT, and provide incentives such as inexpensive computers and laptops. 16 Danang also plans to roll out the public services developed under several procurement packages, extend free WiFi services until 2016, complete the migration of public applications to its Data Center and pilot common data-sharing. Hanoi plans appropriate public information on e-services available via TV and radio, and seeks to deepen the behavioral change in government’s interaction with citizens/business via the Hanoi Portal, as well as inter-agency relationships. The rapidly-changing socio-economic environment in Vietnam means that GSO will need faster and more accurate data on trends and thus strong O&M of sophisticated systems is essential and well-understood. 5. Assessment of Bank and Recipient Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Unsatisfactory 5.1.1 Quality at entry was Unsatisfactory overall. Among positive attributes, the Project was timely and responded to defined country needs, plans and policies. The Bank team sought to ensure that design reflected best practice, global research findings and lessons. Evidence suggests that the internal discussion/review of the proposed operation was candid and wide-ranging. The basic technical approach was sound as the Project could not achieve its objectives without the ICT infrastructure, systems and training financed. Even so, project complexity exceeded the 16 GOV data shows that by end-2012, some 35% of the population was using internet (31.2 million people), 4.78 million were broadband subscribers, and 18.8% of households had computers. 23 country conditions/context and promised too much. The ICR suggests that the preparation team should have contained more experienced ICT operational staff. The Results Framework lacked coherence and was not understood by the clients, who lacked even basic readiness to implement. The operation was immensely ambitious for a first-round ICT project and IDA funding exceeded absorptive capacity. There was inadequate acknowledgment of known risks and calibration of the project to existing human resource capacity, i.e., a schedule which took the learning curve into account. Project scope was broad, especially private sector goals lacking a convincing rationale for their inclusion in a publicly-led project. The institutional/implementation arrangements lacked synergy and created operational difficulties and inefficiencies although it appears that the multi- PIU approach had few alternatives under ODA rules. But this should have prompted analysis/negotiation of mechanisms for cross-PIU collaboration. (b) Quality of Supervision Rating: Moderately Unsatisfactory 5.1.2 The Bank Team conducted 11 supervision missions over a seven year period (based on Aide Memoires and ISRs), well under the Bank’s norm, with uneven spacing of missions (e.g., gaps exceeding 12 months from 2009 to 2012). While this does not take into account regular Bank Procurement and FM supervision out of Hanoi, the impression is that at least until the final two years, supervision from Bank headquarters was uneven. Management recommended immediately post-effectiveness that the inexperienced PIUs receive intensive Bank supervision but missions from headquarters did not increase even though broader forms of guidance were needed, especially for the most recently-constituted PIUs. The MTR accurately diagnosed the issues affecting progress but the continuing rigidity of project design – technical, operational and institutional – and the Bank team’s inability to simplify the Results Framework, improve its coherence and measurability, and focus on development impact, meant that the 2010 Restructuring – a direct outcome of the MTR - had only modest influence on the subsequent pace and quality of implementation. The ICR notes that certain design elements planned at appraisal disappeared early on from the project/supervision agenda: regular consultation with the private sector/SMEs; introduction of demonstration models of e-government; focus on business process change and mobilization of private investment in the ICT sector; and, establishing the planned IT Innovation Fund. 5.1.3 Supervision intensity and quality improved in the final two years, as did the realism of project ratings. The change in Bank task management turned the supervision focus onto diagnosing the main constraints, assigning ICT procurement specialists to the PIUs, and building awareness of development impact. This intensity was costly in terms of supervision budget, as noted in 2.4.6. The ICR suggests that in challenging countries with complex operations, the Country Management Unit (CMU) may need to provide a higher coefficient for supervision. High-level Bank/GOV consultations in 2012 on the Bank’s Vietnam portfolio, and subsequent ministerial directives to PIUs provided critical “weight” to the task team’s efforts which included a second restructuring to inter alia, promote organized project measurement via further adjustments to the Results Framework. 17 Completion of procurement packages linked directly to the PDO became the top priority. Within this finalization scenario, formal impact evaluation had insufficient time and unsuitable institutional conditions for initiation, although Danang and MIC PCU did conduct studies. 17 The 2013 adjustments were the result of an intensive, thoughtful process to link project indicators to the PDO and to reflect what was happening on the ground. However, the excessive number of indicators was not addressed and internal targets for each indicator were increased to ensure the capture of all achievements. See Data Sheet and Annex 2. 24 5.1.4 Fiduciary supervision was satisfactory overall and accompanied the general supervision effort closely because Bank FM and Procurement specialists were located in Hanoi. Procurement supervision was of outstanding quality in the final 18 months, complemented by ICT Procurement Specialists from Bank Headquarters driving the completion of all planned packages. Safeguards supervision was not relevant to this project. 5.1.5 The rating of Moderately Unsatisfactory for Bank supervision quality reflects the overall performance since Effectiveness. The ICR acknowledges the strong- and ultimately successful - effort by the Bank task team working with the PIUs to salvage this project in the final two years. The high cost of supervision (Annex 4) in FY 12-13 testifies to this effort to complete all procurement packages, ensure that installed equipment/systems were operational and oversee the orderly completion of the Project. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 5.1.6 This rating takes into account major weaknesses in the preparation phase affecting quality at entry, the more mixed supervision performance which included well-diagnosed but in practical terms, overly-conservative and unsuccessful efforts to generate progress. While sustained improvement resulted from exceptional measures to salvage the Project in the final years of a seven year operation, this is considered insufficient to raise the rating for overall Bank performance into the Satisfactory range. 5.2 Recipient Performance (a) Government Performance Rating: Moderately Unsatisfactory 5.2.1 Government of Vietnam: Government commitment to the Project was uneven over time and its performance showed significant shortcomings. On the one hand, Government maintained its policy stance on promoting ICT as an engine of modernization and socio-economic development. However: Government dragged its feet in sending the Bank a request for the essential second restructuring despite repeated reminders; its response to high level Bank consultations regarding the Project - within the context of non-performing projects in the Vietnam portfolio – was the result of the Bank’s insistence on action, not its own proactivity, although its directives to participating agencies did have a positive impact on subsequent performance; Government’s last minute request for further extension of the closing date was unjustified; its overall coordination of project activities was deficient; and, implementation of its overall ICT/e- Government Strategy and its Administrative Reform Program was slow with inadequate coordination between the two and effects on the Project including staffing, salaries and skills development. The e-readiness factor was acknowledged in the PAD but project design was not modified, i.e., the Bank shares some responsibility. (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 5.2.2 MIC (PCU and PIU): The Recipient was formally the State Bank of Vietnam with responsibility for project coordination delegated initially to the Ministry of Posts and Telematics (MPT), subsequently changed to the Ministry of Information and Communications (MIC). Pre- MTR, MIC operated with three distinct project teams (PMU, PCU and PIU), a structure not discussed in the PAD. These teams changed twice early on, affecting continuity and the Project’s 25 institutional memory. Ownership of the Project was inconsistent over time. MIC lacked a clear sense of purpose for its Subproject. The Procurement Plan changed repeatedly and new ideas resulted in long and difficult consensus-building. The PCU staff, although large, was inexperienced in project management, did not work proactively with its technical subgroups and had difficulty functioning effectively as the coordinating unit for the entire Project. The technical subcommittees were unable to develop strategic, implementable procurement plans for the Ministry and consultancies were not effectively deployed. Some 18 international study toursMIC delivered a Completion Report of reasonable quality, limited to its own and the PIU/MOFA activities, not the consolidated version agreed before project closing. Competence, leadership, mobilization and coordination issues to varying degrees persisted throughout. Rated: Unsatisfactory. 5.2.3 Hanoi: Several key issues hindered progress in the first stage of the project, including repeated turnover in team membership, lack of qualified staff for procurement, project management and technical aspects, and ineffective use of consultancies. There were protracted delays in mobilizing to effectively implement the Project. Territorial reorganization of the Hanoi City Government further delayed decision-making and obscured project objectives. Approval processes for procurement packages were long and lacked any sense of urgency. Obtaining timely approvals within DIC and from the PPC was a major constraint. Expansion of Hanoi City meant that the development level was uneven city-wide. The PIU sought to focus on rolling out infrastructure services and capacity-building to bridge the digital divide and ensure readiness of users before developing e-applications. Hanoi was the first PIU to complete finalization of bidding documents under the Bank’s new bidding parameters after 2010 and it made reasonable progress in the final years. Rated: Moderately Unsatisfactory. 18 5.2.4 GSO: This agency/PIU performed well, including in the difficult initial years of the Project, with strong disbursement. Positive factors: (i) GSO’s relative autonomy and sense of a shared mission among its 5,000-odd staff (including 64 provincial offices); (ii) direct reporting and access to the Ministry of Planning and Investments; (iii) leadership and commitment of the GSO Director General and management team, especially for using ICT for Change Management; (iv) strength, diligence and technical qualifications of the GSO PIU, although in the final two years the timeliness and quality of some bidding documents was inadequate; and, (v) ability to leverage expertise of international consultants in procurement. GSO planned/executed a good communications and outreach campaign to sensitize GSO staff and the public on the modernization process. It increased the scope and sophistication of additional tools (from the additional reallocation of US$6.2 m in 2010) for data capture/collection, storage, analysis and dissemination. GSO positioned itself as a center of data intelligence and analysis to benefit government, investors, international agencies and citizens, using new technologies for change management, business process re-engineering, and transparent provision of information to clients/stakeholders. It sought to acquire a state of the art statistical information system with business intelligence tools to improve decision-making for policy-makers, investors and development agencies. Rated: Highly Satisfactory 18 The Hanoi PIU, in its comments on the Bank’s draft ICR, strongly disagreed with this rating stating that it felt its performance had greatly improved after the Mid-term Review and merited a Satisfactory. The Bank does not agree with this assessment and retained the rating of MU. See Section 7. 26 5.2.5 Danang City: The Danang PIU performed exceptionally well throughout and the city became a national ICT leader, a remarkable achievement given its modest capacity when formally assessed in 2004. As noted, demand for IDA funds steadily increased to meet the city’s ICT development vision and plans. While disbursement was initially slow, the PIU rapidly laid the foundation for accelerated execution of its Subproject: (i) conducting a city-wide review of government processes; (ii) identifying specific services to be electronically streamlined; (iii) developing a strategy and roadmap for e-government; (iv) effectively training Chief Information Officers (CIOs); (v) making serious efforts to collaborate – especially on training – with other PIUs; and (vi) conducting international study tours. Success resulted from the PIU’s strong, committed leadership, management and technical skills, and clear procurement timetables. It was able to work with specialist consultancies to design/execute sound plans, and had a good grasp of the status and future of ICT infrastructure city-wide. The PIU had the full support of and good relationships with the PPC Leadership (unlike other PIUs) whose high commitment to using ICT for Change Management promoted faster and more efficient approval of procurement packages. Its overall approach was practical, innovation-driven and stakeholder-aware. Danang recognized the technical expertise and management capacity needed to apply the IT EA on a city-wide scale and developed/extended e-government services well beyond what was anticipated at appraisal through skillful use of available (and additional) funds. Rated: Highly Satisfactory. 5.2.6 HCMC: The HCMC Subproject was cancelled by the 2010 Restructuring. The PIU took a mixed loan/grant arrangement but planned outputs (up to the MTR) were implemented with city budget funds, not IDA. HCMC was relatively advanced in its ICT endowment at the time of the 2004 assessment but even so, e-services were receiving an average of only 6,000 requests per month, modest for a city of 8 million. Despite Bank efforts, disbursements of IDA funds were zero at the time of the MTR. The PIU had prepared no procurement or training plans and had no staff assigned to procurement or FM. The main issues affecting HCMC were: (i) DICs’ limited commitment – there was no meeting with the Bank team for two years; (ii) weak leadership of the Subproject and inadequate supervision by the PPC and MIC; (iii) poor implementation capacity, with frequent staff changes resulting in limited understanding of the Subproject approach and Bank procurement and FM procedures; and, (iv) lack of responsiveness to repeated Bank requests about the city’s plans to use IDA funds. The PIU was barely able to absorb its city budget and had no chance of using its IDA allocation. Rated: Unsatisfactory (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory 5.2.9 This rating takes into account the following: (i) Government’s sustained policy stance on ICT-based modernization but moderate shortcomings in terms of the pace and depth of ICT- related policy implementation relevant to the Project, combined with its uneven commitment and responsiveness over time and weak project coordination; (ii) sustained, highly satisfactory performance by GSO and Danang, balanced by uneven performance of Hanoi and unsatisfactory performance of MIC and HCMC. 6. Lessons Learned The complexity of project design needs to be balanced carefully against the client’s human resource endowment and readiness to execute. Scope, scale and scheduling are critical considerations in an innovative, foundational project, especially when the client lacks experience with the Bank. While many Bank projects entail “learning by doing”, capacity must be assessed realistically, and minimum professional and administrative frameworks should be in place before 27 effectiveness to launch activities and build momentum. Front-loading selected, strategic activities into the pre-Board stage and focusing time and budget on human resource development are essential. Practical examples include: having bidding documents ready prior to Board approval as part of the primary feasibility studies to promote better disbursement flow in the initial years; allocating a significant portion of the project budget to change management/initiatives; using Project Preparation Facility (PPF) or Trust Funds to put Enterprise Architecture (EA) in place or conduct EA studies before Board approval to ensure the foundation for subsequent procurement; addressing the issue of ICT and especially Internet penetration to promote rapid evolution of e- services into end-to-end transactions and accelerate citizen uptake; and, greater focus on the human behavioral change element which can directly affect project progress and measurement. Multi-unit institutional arrangements, if unavoidable, demand that the Bank negotiate mechanisms for joint management of essential functions and structured opportunities for collaboration. The objective is to avoid duplication of operational and administrative efforts, inefficiencies, and other well-known “silo” effects. Joint frameworks for procurement, financial management and M&E could have simplified project execution and promoted efficiency without sacrificing human capital formation. This could have been tried in Hanoi where the MIC, GSO and Hanoi PIUs were based. Also, led by Danang and GSO, regular exchanges of experiences and knowledge would have been valuable for Hanoi and MIC. Cancellation would have limited the knowledge gained from implementing this important project but restructuring and extending it in 2010 without addressing design issues was an inefficient use of Bank resources. The ICR concludes that the first restructuring was too conservative and essentially preserved weak design features contributing to implementation delays requiring further restructuring and entailing high supervision costs. The Project needed overall simplification, re-negotiation of the institutional arrangements - at minimum for FM and Procurement – and a marked focus on building leadership. Bank procurement requirements, processes and documents are not optimal for the fast-evolving IT sector and this needs review. The Bank’s portfolio of investments in IT systems and equipment tends to perform poorly, importantly due to procurement constraints. The no-objection process can be slow, and Bank support to inexperienced counterpart teams inconsistent. GSO and Danang PIUs reported delays from one to two years in some packages due to the need for adjustments and re-submission to the Bank. Meanwhile, innovation caused technical specifications to become outdated, contracting could not proceed and further bid evaluations and no objections were required, compounding delay. The Bank needs to revise its IT procurement documents and procedures which have repeatedly proven unsuitable/unwieldy for Bank projects in the technologically fast-moving IT sector. Intervention of senior Bank management with national authorities can galvanize project performance in a failing situation. If the Project team continues to see the operation as worth completing but has been unable to promote progress, high level Bank intervention should be considered. Timing is critical and it may need to occur as a standalone event. Also, the renovation of Bank project task management, bringing in the seasoned ICT specialists essential for projects of this kind and a tighter, more vigorous approach to supervision can effectively salvage an operation, even one with chronic difficulties. ICT projects need to focus on both the supply and demand sides, the latter being an important driver of longer-term sustainability. Plans to further expand online services require attention to user uptake/adoption through repeated communications campaigns tailored to diverse stakeholders and able to solicit feedback to improve e-services quality. The demand side is still weak for many project-supported e-services even in Danang. Citizens prefer traditional “counter- service” accounting for the modest usage of fully transactional online applications even at the end of the Project. That said, many e-services were implemented only in the final year, and demand is also dependent on computer ownership, access and literacy. Finally, the design of a City Portal 28 needs to allow for year-round citizen feedback and engagement functions, with features promoting the smooth transition from manual to digital/e-transactions. The rationale for including weighty private sector goals in a foundational project focused on e-government should be clearly defined. Adequate tools should be provided and a specific strategy devised to achieve and measure goals which include private sector development and improved productivity. Having the public sector take the lead in fostering private sector development through ICT usage was not matched by appropriate tools, limited in practice to training and awareness-building. ICT projects can materially impact the behavior and performance of government, business and citizens and require the same high standards of monitoring and evaluation design and execution as projects in other sectors. Proven evaluation frameworks and methodologies for ICT projects/programs are available. This project demonstrated the implications of a weak M&E framework and inadequate focus on establishing an MIS and a time- bound evaluation strategy and products. Bank-wide by late decade, the focus on M&E quality was much stronger, yet as late as 2012 most PIUs lacked understanding of the PDO/indicators and had not attempted their measurement. The ICR suggests inter alia, that the project team could have obtained an independent, third party, objective review to help re-design the RF at the time of the Level One Restructuring. That said, it is nevertheless unrealistic to expect that the deeper impacts of newly- constructed ICT systems can be measured by end-project. For foundational projects of this type, two indicators – essentially proxies for impact - should be of primary concern: number and type of services online and, numbers of users or uptake of those services. Global experience shows that a minimum of 3-5 years are needed to permit consolidation and second stage developments before impacts such as transparency, user satisfaction, cost reduction and productivity can be measured meaningfully. 7. Comments on Issues Raised by Recipient/Implementing Agencies/Partners (a) Borrower/implementing agencies 7.1 MIC PCU: In response to the Bank’s draft ICR, the MIC PCU sent a letter (see Annex 7) with brief comments, primarily requesting that the Bank reassess its rating for MIC performance on the basis of two reports accompanying the letter (MIC PCU Completion Report and a Project Performance Audit Report (KDiAsia, 2014), designed to provide internal guidance to MIC). The project team reviewed both documents, finding that they accord with the ICR’s representation of the Project’s achievements and experiences. However, the team’s assessment of the overall performance of MIC PCU/PIU was sustained. 7.2 Hanoi PIU: The Hanoi PIU, in its comments on the Bank’s draft ICR, strongly disagreed with the Bank’s performance rating of Moderately Unsatisfactory stating: “Hanoi agrees that there was slow progress in the first stage of the Project. However, since the Mid-term Review in 2010, Hanoi has improved a lot and we cannot deny its achievements in project implementation. Deeply understanding ourselves, Hanoi clearly knows what we can do and what we cannot do within the framework of this Project. That’s why Hanoi decided to drop some packages and focus on the most important and feasible ones. For the remaining packages, we always completed on committed time and the outcomes of these packages are 100% achieved. So we would like to be rated as Satisfactory." The ICR believes the rating of MU is correct. 7.3 Danang PIU: Comments received were editorial and were incorporated. (b) Co-financiers N/A (c) Other partners and stake-holders N/A 29 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components/Activities Estimate (USD (USD millions) Appraisal millions) Subproject 1: ICT Leadership and 21.73 23.17 105.7 MPT Modernization (MPT) Subproject 2: GSO Modernization 13.77 19.59 140.8 Subproject 3: Hanoi E-application 31.20 5.35 17.2 Subproject 4: Danang E- 17.33 27.56 157.3 application Subproject 5: HCMC E- 10.61 0.02 0.1 application Subproject 6: Project 1.50 1.39 84.5 Coordination Total Baseline Cost 96.29 77.08 79.0 Physical and Price Contingencies 4.84 19 0.00 0.00 Recurrent costs 5.83 0.00 0.00 Total Project Costs 106.96 77.08 71.0 Front-end fee PPF 0.00 0.00 0.00 Front-end fee IBRD 0.00 0.00 0.00 Total Financing Required 106.96 77.08 71.0 **Aggregate total US$13.48 million cancelled (b) Financing Appraisal Actual/Latest Type of Co- Estimate Estimate Percentage of Source of Funds financing (USD (USD Appraisal millions) millions) Recipient 13.24 2.96 20 22.36 International Development 93.86 21 74.12 22 78.97 Association (IDA) 19 Price and physical contingencies are shown in aggregate in project documents 20 This is a best estimate provided by the Recipient as at June 26, 2014. 21 Amount of US$93.86 m is shown in Bank system as “original signed amount” as at June 26, 2014. 22 Original commitment of US$93.86 m; cancellation US$13.52 m; total disbursed US$74.12 m; undisbursed balance US$6.23 m. All data from Bank system as at June 26, 2014. 30 Annex 2. Outputs by Component This Annex comprises the following: (A) glossary of ICT terminology used by the project and in the ICR; (B) matrix of component outputs by Subproject; (C) matrix showing original and revised Project Outcome Indicators and Intermediate Outcome Indicators (Results Frameworks); and (D) completion status of all procurement “packages” financed by the Project. A. Glossary of terminology used by the Project and ICR Term Technical/Common Meaning Interoperability Standards and Means to enable public administrations to Framework collaborate; framework for ICT-enabled governance; not only a means of improving public services but of enabling multi-level governance; supports information sharing among ICT-enabled administrations including across borders. (Wikipedia) Enterprise Architecture High-level strategic technique to help senior managers achieve business and organizational change; a “coherent set/whole of principles, methods and models used in the design and realization of an enterprise’s organizational structure, business processes, information systems and infrastructure (Lankhorst et al, 2005). Portal/City Portal A specially-designed web-page at a website which brings information together from diverse sources in a uniform way. Usually, each information source gets its dedicated area on the page for displaying information; the user can configure which ones to display. (Wikipedia) e-Service Role of technology in facilitating delivery of services. “…. deeds, efforts or performances whose delivery is mediated by IT including e-tailing, customer support and service delivery”. Three main components: service provider, service receiver, and service delivery (technology). (Wikipedia) e-Government “Digital interactions between a government and citizens, government and business/commerce, government and employees, and also government and government/agencies”. (Wikipedia) National IT Architecture Organized set of consensus decisions on policies and principles, services and common solutions, standards and guidelines as well as specific vendor products used by IT providers both inside and outside an IT entity. IT architecture is designed to plan, acquire, build, modify, interface and deploy IT resources throughout departments. – identifying technologies which work together to satisfy user needs. (Wikipedia). G2B (Government to Business) G2B is the online, non-commercial interaction between local and central government and the 31 commercial business sector rather than private individuals with purpose of providing businesses with information and advice on e-business best practices. G2C (Government to G2C refers to a communications link between a Consumer/Citizen) government agency and the general public. Most often refers to what takes place through ICR technologies but can also include direct mail and media companies. G2C can occur at the federal, state or local levels. (Wikipedia) G2G (Government to Government) G2G is the electronic sharing of data and/or information systems between government agencies, departments or organizations. The goal is to support e-Government initiatives by improving communication, data access and data sharing. (Wikipedia). Front and back-end solutions Terms used to characterize program interfaces and services relative to the initial user of both. User may be human being or program. Front end – users interact directly. Back end applications or programs serve indirectly to support front end by being closer to the resource or having capacity to communicate with the required resource. Meta-data Standard Requirements intended to establish a common understanding of the meanings or semantics of the data to ensure correct and proper use and ensure correct and proper use and interpretation of data by its owners or users. It is “data about data”. Structuring information that describes, explains, locates or otherwise makes it easier to retrieve, use, or manage and information source especially in a distribution network environment, e.g., internet or an organization. (Wikipedia) Transaction Centers City-level facilities for private sector (esp. SMEs) to obtain hands-on training, view exhibits, and get exposure to new tools and technologies, highly- specialized training and business applications. Tier 1 Applications In project context (Hanoi): Integration/linkage of departmental portals and websites to the Hanoi City Portal Tier 2 Applications In project context (Hanoi): Integration of public services for citizens via Internet. Tier 3 Applications In project context (Hanoi): E-government services processed at One Gate Service (business registration; reissuance of birth certificate; temporary construction license; new car registration etc) Tier 4 Applications In project context (Hanoi): E-government services which require online payment (declaration and payment of income taxes; customs declaration etc). Semantics Semantic technologies are often proposed as 32 important components of complex, cross- jurisdictional, heterogeneous, dynamic information systems (CSIRO). Organizational Interoperability Modeling organizational processes, aligning information architecture with organizational goals and helping these processes cooperate via ICT. Has specific relevance to e-government, i.e., not only technology but a broad concept of modernization of public administration including new skills and cultures. (Kubicek, Cimander and Scholl). Telecom Sector Roadmap Technology/telecom roadmap is a plan that matches short-term and long-term goals with specific technology solutions to help meet those goals. Three major uses: helps reach consensus on a set of needs and technologies required to satisfy them; provides a mechanism to help forecast technology developments; and, provides a framework to help plan and coordinate technological developments. LAN Local Area Network WAN Wide Area Network MAN Metropolitan Area Network Metronet Fiber-optic network delivering high-speed Internet CERT Computer Emergency Response Team (for Internet) for computer security incidents Root Certificate Authority In ICT computer security, a root certificate is either an unsigned public key certificate or a self-signed certificate that identifies the Root Certificate Authority as part of a public key infrastructure scheme. The most common commercial variety is a digital signature from a Certification Authority (CA). eGovFrame (Korea) E-Government Standard Framework developed in Korea. Is an open-source software development framework “eco-system” which ensures interoperability needed for information-sharing and offers standardization and re-usability. Facilitates development of applications based on open standards, resulting in improved software quality, more efficient development, and greater re-use of software modules. Application Architecture Framework of strategies and plans for developing e- services. Not solely about ICT infrastructure to build online services but sets out all the structures – business processes as well as technology – for effective e-services. 33 B. Outputs by Component The Project comprised five “subprojects” with internal “components” in each case, and covering three development themes: (i) ICT leadership and modernization of the MPT; (ii) modernization of the GSO; and (iii) e-applications for the Municipalities of Hanoi, Danang and Ho Chi Minh City (HCMC). Changes were made to some components by the 2010 Restructuring including canceling the HCMC subproject altogether. Subproject 1: ICT Leadership and MPT (MIC) Modernization This subproject sought to strengthen the capacity of the Ministry of Posts and Telematics in development, implementation and oversight of Government’s national ICT strategy and policies; strengthen its capacity to formulate telecommunications and ICT policy; and, improve its operating efficiency and effectiveness. The ICT capability of other government ministries would also be improved via the function of the CIO (Chief Information Officer). There were six components: Development of e-Government capability in MPT; Creating an MPT Flagship Portal; Creating a center for training, improving and managing CIOs; Training in ICT Policy and Regulation; IT and Telecommunications Regulation; and, IT Architecture and Interoperability Framework for e- Government. The ICR notes that the original description of the components in the PAD, Annex 4 shows Subproject 1 with 6 components and 32 individual activities, many of which show no clear link as outputs supporting the Subproject-based Intermediate Outcome Indicators in the Results Framework. This list was modified by the 2010 Restructuring and the gap with the also-revised RF seems to have widened. Component 1.1: Development of e-Government capability in MPT (changed to MIC) Level of Planned Actual Attainment 1. Review and upgrade MPT’s (MIC’s) current ICT MIC’s ICT infrastructure was reviewed and future needs Achieved infrastructure and assess future needs to provide e- determined. E-government framework, policy and standards set Government services up. Enterprise Architecture and interoperability standard formulated. E-government framework and security completed. Upgrades included: LAN network in MIC and 12 DICs with user acceptance tested and end-user training conducted. IT equipment for smart building communications system installed between MIC and 4 agencies (Regional MIC offices in Danang and HCMC, Telecom Authority in Hanoi, and MOFA). Video- 34 conferencing, IP phone and building security system installed and operational. 2. Assess and upgrade the existing DPT (changed to DIC) PCs, laptops and printers, email, video-conferencing and Achieved ICT infrastructure LAN/WAN acquired/installed for 12 DICs (District Offices of the Ministry of Information and Communication). In 12 DICs, 83 LANs built, IP phone system connects all departments and 4 MC departments outside. 3. Re-engineer governmental process flows to facilitate and Video-conferencing system is facilitating more frequent and Achieved increase interaction within MPT and with related efficient communications between MIC and DICs; standardized government entities such as DPT (DIC) email system also implemented. 4. Create an e-library to incorporate current and new e-Library not established. Activity dropped. Not achieved documentation for the MPT. Provide other ministries the opportunity to access the e-library. 5. Installation of intelligent system and united New component (2010). Intelligent/smart building system was Achieved communications installed at MIC HQ in Hanoi and MIC Data Center in Hanoi. United communications via email, VC, LAN/WAN installation. 6. Design and manage pilot programs for two to three Reduced from 7 to 3 in 2010. MIC implemented 3 licensing e- Achieved informational e-services services/transaction launched in May 2013: radio spectrum, publications, and registration of imported publication portfolio for publishing companies. Paper-based applications no longer accepted. 7. Develop complementary informational, interactive and Internal email system, Video Conferencing system and MIC Achieved transactional services Portal developed. 8. Combine pilot project experience and develop additional Also, visa online application system installed in 95 visa offices Achieved e-services worldwide – piloted in September 2013 and now fully- deployed. 9. Establish in-house or outsource management of central Achieved through the MIC overall ICT upgrade. Central Achieved infrastructure, including gateway, portal and content infrastructure (Gateway, portal and content) established, and managed by combination of in-house and outsourced professionals. 10. Designing a Public Key Infrastructure (PKI) scheme to Activity dropped Not achieved support e-government and develop a national e- authentication framework 35 11. Implementation of the upgrade of the Root Certificate Root Certification Authority completed and end-users trained Achieved Authority (CA) 12. Technical assistance on e-government framework and Technical assistance om e-government framework and IT Achieved IT applications standards applications standards implemented. 13. National Computer Emergency Readiness Team Removed by 2010 Restructuring Dropped (CERT) 14. Building a test center to evaluate IT products/solutions New (2010). Completed November 2013. Center used for Achieved testing software from Yen Bai province and e-office. 15. Consultancy on solution and implementation plan for e- New (2010). Consultant contracted and plans for e-document Achieved document interchange system and on solution and inter-change and integrated email system developed. implementation plan for integrated e-mail system 16. Upgrade of ICT infrastructure for MIC and selected See #1 and #2 above Achieved DICs 17. e-Gateway for MIC E-Gateway (phase 2 of MIC’s Portal) completed and Achieved operational (www.mic.gov.vn) 18. Technical assistance on telecommunications and ICT TA contracted and supplier-delivered for telecom. 50 MIC Partially policy and regulations and on sector monitoring and staff trained in ICT policy and regulations. Modest progress on achieved evaluation system sector M&E. 1.2: Creating an MPT (MIC) Flagship Portal 19. Design the enhancement and expansion of the existing Enhancement and expansion of MIC website designed. Achieved MPT (MIC) website 20. Develop and activate MPT (MIC) informational portal MIC Informational Portal developed and activated Achieved 21. Develop and activate MPT (MIC) interaction portal MPT/MIC flagship Portal established. No information re Partially including complaints management framework complaints management achieved 22. Develop and activate MPT (MIC) Transactions Portal MIC Transactions Portal developed and activated Achieved 1.3 Creating a center for training, improving and managing CIOs Selected items to be supported: 23. Design CIOs and IOs job descriptions and CIOs and IOs job descriptions and responsibilities designed. Achieved responsibilities 24. Design and conduct CIO and IO training for about 400 MIC conducted CIO training in Singapore and also invited Achieved CIOs international trainers to Vietnam to conduct larger scale training. 497 CIOs/IOs trained (277 from Ministries, 150 from state-owned enterprises, 70 from SMEs). 36 25. Develop the next IT Master Plan (2006-2011) and assist IT Master Plan developed. M&E of National ICT Strategy not Partially M&E of the National ICT Strategy established. achieved 26. Establish awareness-raising ICT program and design ICT awareness-raising programs established for stakeholders Partially National Roadmap via public and private associations, including citizens and SMEs. National Roadmap not designed. achieved NGOs and academia 1.4 Training in ICT Policy and Regulation 27. Develop and design ICT literacy and technical training ICT literacy and technical training designed/delivered. Actual Partially for MPT (MIC) and DPT (DIC) staff. Increase ICT literacy increase in ICT literacy at middle and top management levels is achieved of MPT (MIC) staff and DPT’s (DIC) middle and top likely to have increased given type of training but was not management. measured. 28. Develop and provide telecommunications policy Telecommunications regulatory and ownership issues were part Achieved training of MPT (MIC) and selected staff of DPT (DIC) – of telecom policy training for MIC and DIC staff. telecommunications regulatory and ownership issues. 29. Develop and provide training for top management and 497 CIOs and IOs trained in diverse aspects of ICT including Achieved executives of MPT (10-20% MPT (MIC) staff) on economic integration, telecom law, trade agreements etc. international economic and economic integration, telecom law, international treaties and trade agreements with foreign governments, and the functions and responsibilities of international organizations in ICT. 30. Provide MPT (MIC) top/middle management (30-40% 497 CIOs and IOs trained in diverse aspects of ICT including Achieved staff) and top management of DPT (DIC, 10-20% staff) leadership, strategy and policy-making with ITC HR training – leadership, time management, strategy and policy-making etc. 31. Organize international study tours and training trips. 18 international study tours conducted to Europe, USA, China, Achieved japan, Australia and Korea. Training in ICT services development and delivery, spectrum management, software outsourcing, information security, database development, and knowledge exchange. 1.5 IT and Telecommunications Regulation 32. Assess Vietnamese telecommunications regulations Partial assessments conducted Partially and policies – goal was to gradually adapt Vietnamese achieved telecom regulations to international standards. 33. Address telecom regulatory and policy issues on a per 37 needs basis. Conduct overview training for key MPT staff on selected topics and facilitate knowledge transfer. 34. Review international best practice operations guidelines This activity was removed from the project in 2010 but Achieved and responsibilities of a CERT – ownership, organizational completed by government with own funding. outside the structure and security, including of existing systems. project 35. Review international best practice framework, International best practices reviewed and CA Authorities Achieved operational guidelines and responsibilities of Certification assessed. Root Certification and Test Center 100% completed Authorities. Provide applied standards to set up a national by end-2013 (MIC), and end-users trained. Systems for digital certification authority to issue/verify digital certificates. certificates and signature developed and transferred. 36. Review international best practices for setting up four See #35 above. Achieved public key infrastructure main entities – certification authority, registration authority, certificate directories, and certificate management system. 1.6: IT Architecture and Interoperability Framework for e-Government 37. Review international experience and best practices International experience and best practices reviewed Achieved 38. Review existing plans and documentation in Vietnam, Existing plans reviewed. MIC completed National IT Master Achieved including National IT Master Plan and draft e-Govt Plan; Plan. Hanoi and Danang completed city-level IT Architecture city-level information about IT architecture in Hanoi, and IT Master Plan. Danang and HCMC and its IT Master Plan. 39. Consult with key stakeholders and user groups involved Involved user groups and key stakeholders consulted on Achieved in national and city level IT and e-Government strategy and national and city-level IT and e-Government strategy and implementation implementation 40. Identify high level options for IT strategy, architecture, High levels options partially identified by end-2013 – mainly Partially roadmap and specifications IT strategy and architecture achieved 41. Develop a document on the common IT architecture Document developed. See #1 above. Achieved and inter-operability framework for Vietnam 42. Set up standards for IT systems Standards for IT systems established Achieved 1.7: MOFA under MIC 43. Data Center Added 2010. Data Center completed and operational Achieved 44. Implementation of online visa services Added 2010. Online visa application service established and Achieved functional in 95 countries. Subproject 2: GSO Modernization 38 This subproject was to support implementation of a strategic plan for modernizing GSO including improving the quality, transparency and timeliness of socio-economic and poverty-related data and information to fully meet the demands of government agencies, research institutions, and other data users as the basis for policy planning and decision-making; to modernize the system to reach and advanced level in the region; and, to improve data flow, statistical analysis and dissemination of useful data and information. There were three components. Outputs were as follows: 2.1: ICT Infrastructure Upgrade at GSO Level of Planned Actual Attainment 45. Review, streamline and automate internal workflows Achieved through ICT infrastructure upgrade. GSO Enterprise Achieved within GSO and between GSO and PSOs to facilitate Architecture in place and all IT initiatives at GSO required to information dissemination, workflows and processes, and follow this standard. Information dissemination, workflows and create efficiency. Devise and implement an appropriate processes facilitated by: VPN connection between GSO HQ, all architecture and strategy. PSOs and 3 ITCs using MPLS and public internet connection; new email system used throughout GSO ensuring secure document exchanges. Network connectivity for GSO Enterprise Applications improved/more secure. 46. Assess and define future needs for GSO’s networks, Achieved through ICT infrastructure upgrade. Network Achieved strengthen network security and review needed hardware security improved (see above). Hardware and software options (databases) and software (GIS) options. reviewed. Gap bridged between GSO plans and IT for centralized and coordinated IT services completed via SSIC and SHS; effective IT and priority-setting decisions based on project activities. 47. Define course standards and design courses to train Course standards defined and courses designed for GSO Achieved GSO employees. employees. Training conducted/completed for GSO staff on new sector databases, database standardization and integration, network administration, and SSIC system. GSO statisticians trained in MS Office; IT experts trained in SQL/Sharepoint etc. Total 690 GSO staff trained including at provincial level. 2.2: Data Flow and Statistical Analysis Enhancement through ICT 48. Review methods of data collection and processing, Completed implementation of System of Statistical Information Achieved assess and upgrade PSOs’ and selected district statistical Collection (SSIC) in GSO HQ in Hanoi, and three IT Centers in offices’ ICT equipment and provide relevant training in Hanoi, Danang and HCMC. User training conducted. 3 out of 39 data collection and analysis 9 planned databases developed (population, SME and BDS); data standardization and integration methodology developed; population reports automatically generated; SSIC system implemented at GSO HQ and 3 ITCs. End-user training conducted. 49. Re-design GSO’s questionnaires and introduce an Achieved with significant results. Questionnaires re-designed. Achieved optical scanning system to make data entry more efficient. Optical scanning infrastructure introduced. Efficiency gains: Establish system at HQ and train employees for 2009 Data collection and processing for 2009 Population Census Population Census. took 8 months (33% time reduction); for 2011 Agricultural and Rural Census (50% improvement in overall processing time, from 2 years to 1); and for CPI survey (data collection and processing time reduced 33% - from 15 to 10 days). 50. Assess and upgrade ICT infrastructure at PSO and DSO All PSOs and ITCs connected with GSO HQ via VPN using Achieved levels. MPLS and public internet. Data transfer between GSO and PSOs improved. Network connectivity for GSO Enterprise Applications. 51. Develop and provide necessary ICT equipment training 122 CIOs trained and certified; 83 GSO staff and IT experts Achieved at PSO level; develop and provide statistical analysis from 3 IT Centers trained in Network Administration; 395 GSO training for GSO and PSOs to ensure use of internationally and 63 PSO staff/statisticians trained in MS Office; 27 IT accepted analytical tools. experts in 3 IT Centers trained in SQL/Share Point/New Window Server. Total 690 trained. 52. GIS application support mapping administration field GSO established field office as specified Achieved office. 2.3: GSO Portal and Intranet Creation 53. Re-design and development of architecture and GSO Intranet Website and Portal re-designed including Achieved functionalities of existing GSO Intranet Website and Portal architecture and functionality. New portal, Statistics Hub System (SHS) information dissemination portal was implemented and is operational 54. Facilitate GSO internal processes, workflows and Achieved through the Statistical Hub. It is set up in-house Achieved procedures through the Intranet and Portal. (GSO Data Center in Hanoi). All GSO statistical data domains have integrated with GSO Portal and basic statistical data is provided to the public free of charge. 50% of GSO data have 40 been integrated into Statistical Data Warehouse. 55. Make accessible on the Internet more accurate and Achieved through the new Portal. GSO plans to conduct Achieved recent country statistics, statistical analyses and findings on regular update of information. Vietnam which are (were) delivered in hard copy format. Update information more regularly for better targeted and informed policies. Subprojects 3, 4 and 5: E-Government and E-Development: Subprojects for Hanoi, Danang and Ho Chi Minh City e-Government had similar objectives and components, supporting activities to enhance the ICT environment in these cities, and providing the basic foundation for deployment of e-Government in their district offices and departments. They were expected to facilitate the development of various types of e-government services, e-applications for business, and development of ICT skills. These were expected to have a positive impact on government accountability, effectiveness, transparency, access to information by citizens and the business community, citizen and business interactions, and public administration reform efforts. The competitiveness of SMEs was expected to improve, with positive spillover effects on their trade opportunities. Despite broad similarities, individual subproject design reflected the different status of ICT development in each city and their particular application needs. Outputs were as follows: Subproject 3: Hanoi e-Government Level of 3.1 Identify Priorities, Key Future Requirements and Develop a Strategy Attainment Planned Actual 56. Assess key ICT infrastructure and applications, analyze Key ICT infrastructure and applications assessed as well as Achieved and evaluate current level and demand level and demand. 57. Identify priority business processes/services 14 priorities identified: Dept Planning and Investment (issuing Achieved investment certificates to firms); Dept. Industry and Trade (issuing certificates to establish branch representative offices to foreign firm); Hanoi Police (registration of car and motorbike); Dept. of Health (grant certificate to practice medicine); Dept. Transportation (grant/change driver’s license); Dept. Information and Communications (grant certificate for importing non-business printed publications); Dept. Construction (construction permits); Districts (grant food safety certificates); Dept. Health (issue certificates for food 41 standards); Dept. Justice (issuing judicial history); Districts (grant construction license); District (provide land information from cadastral records); Districts (issue certificate of housing and land use and ownership rights). 58. Develop clear and comprehensive e-Government Strategy, Roadmap, city-level IT Architecture Framework and Achieved strategy, roadmap and city-level IT architecture for e- IT Standards were developed for Hanoi e-Government. Government in Hanoi. Also covered will be specific issues such as portal layout and taxonomy, operating systems, languages, approach to applications, security, interpretation, standards etc. 3.2: Local e-Government Enablement 59. Upgrade and expand ICT infrastructure Deleted from the project in 2010 restructuring due to self- Cancelled funding by government 60. Develop and implement core elements of the shared IT Added in 2010. Core elements of shared IT architecture Achieved architecture including Data Center developed. EA Master Plan designed for 3 levels of government (Department, Ward and Commune). City-level architecture designed. Aligned IT architecture designed for 57 Departments. Data Center completed and operational. 61. Develop, expand and implement gateway and city Data Center to host City Portal completed and related software Achieved portal, automate information, and provide basic municipal installed. 18 Department Portals and 16 applications integrated information and services via the portal. This would include with Hanoi City Data Center. One-stop software for some of the over 200 services provided by Hanoi’s government agencies integrated in Data Center. Easier, more municipal government, as well as information about these reliable access to government information. Public services services and relevant procedures. integrated via internet (20 services at 2nd level, 8 services at 3rd level, 2 services at 4th level). 62. Manage portal and gateway infrastructure Portal and gateway infrastructure located in the Hanoi Data Achieved Center and managed there. 63. Implement process improvements and integrate key 57 departmental portals and websites integrated. Public Achieved departments and offices services integrated via internet (see # 61). 64. Validate and/or develop business model for new ICT Originally included in HAN 9 package but after consideration, Cancelled centers was dropped 65. Establish and equip ICT and social/shared facilities in Dropped because Hanoi decided to use local funds for its Cancelled the new ICT Transaction Center implementation. (ICT and social/shared facilities in new ICT 42 Transaction Center were established and equipped using own funds). 3.3: Implementation of e-Applications (and transaction center) 66. Implement e-procurement application involving 3-5 Dropped because already implemented by a project of the Cancelled. different departments: provide e-procurement information Ministry of Planning and Investment. via an e-procurement portal to businesses (stage 1). 67. Implement online land and population management E-services developed after definition as business priority. See Achieved registration processes # 57 68. Implement e-applications, e.g., driving licenses and E-services developed after definition as business priority. See Achieved representative office permits # 57 69. Further develop direct (offline) and online services for 20 public services integrated via internet – easier, more reliable Achieved the public, government and businesses access to government information 70. Develop GIS and GIS-related e-services This is planned to be implemented under a proposed 2nd phase Cancelled project 71. Develop new/alternative access channels, such as This is planned to be implemented under a proposed 2nd phase Cancelled kiosks or online booths (e.g., in post offices) through which project the city’s portal and e-government services can be accessed by citizens and business people 3.4: ICT Human Resource Development and Awareness-Building 72. Develop a Master Plan for ICT human resource This was cancelled due to the availability of government Cancelled development in Hanoi and design a training program for funding. Government completed the study by itself. the next five years. Facilitate and conduct consultations to define priority areas, select pilot locations, delivery modalities, standards and other program cornerstones. 73. Upgrade existing training facilities and equip the See # 72 Cancelled training centers with ICT equipment and supplementary facilities required to deliver training courses. 74. Improve and expand existing training capabilities and See # 72 Cancelled competencies: selected foreign and local specialists provide “train the trainer” program in specific ICT-related and e- government areas. 75. Develop training material (software, text books, This activity was cancelled by the 2013 Restructuring (HAN 9). Cancelled computer-based training, online and multimedia) and It does not include implementation of the Master Plan or 43 databases, coordinate with ICT training facilities. training program itself and is not related directly to PDO achievement. 76. Provide general ICT training and awareness programs 30 government leaders trained in Singapore in advanced ICT, Partially for government officials in advanced ICT, good governance, project management /other via 5-day course: “E- achieved governance, customer orientation/communication, desktop Government towards Public Service Excellence” organized by and functional applications and project management Civil Service Collage, Singapore. Target was 340 leaders and (management by objectives, M&E). key officials trained. Also, 100 SMEs trained in CIO but using local funds. 77. Provide specialized ICT and ICT-related training on See # 76 Partially specific topics for all three target groups but esp. achieved government and the business community. Subproject 4: Danang E-Government 4.1: Identify Priorities, Key Future Requirements and Develop a Strategy 78. Assess key ICT infrastructure and applications, analyze Achieved and evaluate current level and demand 79. Identify priority business processes/services and Priority business processes and services identified. Based on Achieved departments/districts: identify which core processes, this assessment, Danang PIU implemented 21 e-services (e- information and services should be e-enabled and prioritize government platforms). within relevant departments 80. Develop a clear and comprehensive e-Government Danang e-government strategy, roadmap and city-level Achieved strategy, roadmap, and city-level IT-architecture for e- Enterprise Architecture (EA) in place. government in Danang. 4.2: Local e-Government Enablement 81. Upgrade and expand ICT infrastructure Target was extended in 2010 Restructuring and exceeded the Achieved revised target. Danang Metro Area Network (MAN) completed; public WiFi completed; 165 public WiFi access points available to citizens free of charge until pricing model introduced; Danang Data Center completed and operational; e- Learning portal completed and operational. 82. Develop and implement the core elements of shared IT Core elements/facilities developed and implemented. Security Achieved architecture, including core facilities (data center and systems and interoperability standards/systems and back-up administrative systems). Develop and set up security implemented. See Danang study (Leading Associates/KDiAsia (online and offline) and interoperability standards and 2014) on Danang security. 44 systems, and back-up systems. 83. Develop and expand city portal, automate information Contact Center established providing information on public Achieved flow, and provide basic municipal information and services services and social/economic issues; Danang MAN connecting by portal 95 Departments/agencies, Communes and Wards; 32 Internet Stations installed in Communes/Wards where such services limited; 10 information kiosks for tourists; 165 public WiFi access points installed and additional points commissioned. 84. Implement workflow solutions and integrate key 90 departments/agencies provided with video phones and IP Achieved departments and offices: implement standardized and phones; video conference service established; Business Process integrated workflow solutions and applications for all key database developed and commissioned; MIS departments (3-5) and district offices developed/operational; e-Wards/e-Districts completed/commissioned; 100% of one-stop shop used by Wards and Districts connected to MAN and transferring data to Department of Home Affairs. All identified workflows and reporting lines automated. 85. Validate and/or develop business models for ICT Business models validated, plans developed and Achieved transaction center: review plans for ICT transaction center reviewed/revised and best practices incorporated. Based on and make required revisions. Incorporate best practice or this, ICT Transaction Center was developed and is now innovative ideas from other countries. operational. 86. Set up ICT and social/shared facilities (administration ICT and social/shared facilities established in Danang ICT Achieved rooms and offices, conference center, training facilities, Transaction Center serviced offices, data center, meeting and training rooms) in ICT Transaction Center 87. Manage portal and gateway infrastructure (outsource or Data Center was selected (construction of data center was Achieved joint management) added by 2010 Restructuring), completed and is operational and under management both jointly and via out-sourcing when needed. 4.3: Implementation of E-Applications and Transaction Center 88. Implement e-procurement application involving 3-5 E-Procurement application was cancelled by project but Cancelled different departments: Provide e-procurement information implemented with government’s own funds. via an e-procurement portal to businesses. 89. Implement e-business registration/licensing e- Business Registration System and Investment Project Achieved application: implement an online business registration and Management System completed development phase and 45 licensing process integrated with the gateway and city launched/operational in January 2014. Integrated with the portal. Gateway and City Portal. 90. Implement 2-3 additional e-services such as drivers’ Total 21 e-services launched and operational including: e- Achieved license registration Drivers’ License system; e-Construction Permits system; Public Transport Management System commissioned with 100% of buses equipped with GPS, managed and monitored centrally by DOT based on GIS map; Water Monitoring System operational with raw water monitored and controlled 24/7 based on best practice criteria before treatment and processing. 91. Further develop direct and online services by ICT See #90 above. Achieved Transaction Center including equipment and services 92. Develop e-applications and e-commerce SMEs sell-informed about services provided by the Contact Achieved market/environment for SMEs: promote local software Center; e-Learning Portal launched with 400 courses available industry by developing/setting up standard applications, e- in ICT and soft-skills – 15 modules targeted specifically to markets etc. Danang should link to USAID’s software SMEs. Software developers trained in Project Requirement cluster development projects in Hanoi and HCMC Management; 5- software developers from SMEs joined e- Government Platform courses. Basic IT Knowledge program televised 3 times/week (68 sessions available). Program contents uploaded to website (www.phobienkienthuc.dnict.vn) and 27000 viewer hits by end-2013 (about 404 viewers average per/lesson. Danang Training center conducted 27 training courses with 1,184 participants of which 773 online. 93. Develop alternative/new access channels such as kiosks Danang PIU developed Danang City Wi-Fi and mobile apps for Achieved or online booths (e.g. in post offices) through which the the City Portal and e-Government services, more convenient city’s portal and e-government services can be accessed. and accessible 24/7. 4.4: ICT Human Resource Development and Awareness Building 94. Develop a Master Plan for ICT human resource Danang’s ICT HR Development Master Plan was developed Achieved development in Danang and design a training program for and approved by People’s Committee. the next 3-5 years. Define priority areas, select pilot areas (locations), delivery modalities, KPIs, standards and other program cornerstones 95. Upgrade and/or expand ICT training programs in Danang PIU developed ICT training modules for SME and Partially educational institutions; implement ICT infrastructure for private sector. SMEs well-informed about services provided by achieved 46 ICT training-oriented programs at educational institutions Contact Center. 268 SMEs took e-learning courses and over in Danang. Develop training programs and applications for 100 took basic IT training courses. Information on ICT use in classrooms. programs in educational institutions/classrooms not available. 96. Expand facilities of training and applied research Training and applied research facilities set up in Danang Achieved centers (in ICT Center): set up and equip a training center Transaction Center. and applied research center with supplementary facilities (labs, conference, broad band access, online and physical library) within the Transaction Center. 97. Improve and expand existing training capabilities and Training competencies upgraded. Foreign and local specialists Achieved competencies: selected foreign and local specialists, “train contracted under the Project to train selected IT specialists in the trainer” program in specific areas Danang. 98. Develop training programs and materials (software, Danang PIU developed online e-Learning modules. See # 100. Achieved textbooks, online and multimedia) and databases on various Also, training sessions were broadcast by local TV station with topics, coordinate with ICT training facilities recorded sessions uploaded to City Portal. 99. Provide general ICT training and awareness programs 497 CIOs and IOs trained (277 from Ministries and 63 DICs, Achieved for government officials in advanced ICT, good 220 CIOs/IOs from state enterprises). See # 100 governance, customer orientation/communication, desktop and functional applications, and project management 100. Provide specialized ICT and ICT-related training in e-Learning Portal operational (http://multimedia-lab.org); 400 Achieved specific topics for all three target groups, especially courses in ICT and soft skills available in e-Learning portal; 75 government and business. software developers trained in Software Engineering, Software Project Management, and Software Testing; 27 government leaders received CIO training; 25 engineers trained in Integrated Circuit Design. Subproject 5: Ho Chi Minh City E-Government 5.1: Identify Priorities, Key future Requirements and Develop a Strategy 101. Identify priority business processes/services and Government-funded. Cancelled departments/districts: identify which core processes, information and services should be e-enabled and prioritize these within relevant departments 102. Develop clear and comprehensive e-government Same Cancelled strategy, roadmap, and city-level IT architecture for e- government in HCMC 47 5.2: Set up ICT and Social/Shared Facilities in New ICT Transaction Center 103. Review plans for ICT Transaction Center (terms, Same Cancelled services, facilities, business model etc) and make any required revisions. Incorporate best practice or innovative ideas from other countries 104. Equip facilities such as social, shared, and Same Cancelled administration rooms and offices, conference center, meeting and training rooms 105. Develop and set up security (online and offline) and Same Cancelled interoperability standards and systems. Implement middleware, data warehouse, or central databases required for shared services under the IT architecture 106. Develop, expand and implement gateway and city Same Cancelled portal, automate information, and provide basic municipal information and services via portal 107. Implement standardized and integrated workflow Same Cancelled solutions and applications for all key departments (up to 5) in district offices. 5.3: E-Application, E-Commerce 108. Provide e-procurement information via an e- Same Cancelled procurement portal to business 109. Implement 2-3 e-applications in the areas of vehicle Same Cancelled registration, driving licenses, representative office licensing. Further develop DPI’s online business registration and licensing process, integrate it with other departments to improve the online process and link with gateway and city portal 5.4: ICT Human Resource Development and Awareness Building 110. Develop a Master Plan for ICT human resource Same Cancelled development in HCMC and design a training program for the next 5 years. Identify priority areas, select pilot locations, delivery modalities, standards and other program components 48 111. Select foreign and local specialists to provide “train Same Cancelled the trainer” program in specific areas related to ICT and e- government. Coordinate with MPT on CIO/IO training program (PIU/city-level training, MPT/national training). Provide training for ICT Transaction Center Staff. 112. Develop training programs and materials (software, Same Cancelled textbooks, online and multimedia) on various required topics, develop databases. Test these in small groups and refine based on feedback 49 C. Results Framework: Original (PAD) and Restructured (2010 and 2013) Note: Indicators are listed as an autonomous “set” resulting from each Restructuring. Revisions, deletions and/or additions are not aligned with previous versions. See Data Sheet for details on adjustments over time to individual indicators. Original (PAD) Revised (2010 Restructuring) Adjusted (2013 Restructuring) Project Results Indicators Increase in share of ICT sector in GDP by 2 Improved formulation and implementation of Formulation and implementation of ICT policies and percent ICT policies and common standards common standards in MIC (R) Increase in ICT capacity and knowledge in Reduced transaction costs, increased Upgrading ICT infrastructure to support e- respective government departments transparency and users’ satisfaction of government technical capacity in MIC, GSO, selected government services through e- MOFA, Hanoi and Danang (N) 15 percent use ICT for business operations in application 2010 and improve productivity Reduction of time required for one transaction of G2B services made available online selected government services through e-applications Domestic IT service sector grows by an average More private firms in selected cities use ICT (Hanoi, Danang) (R) of 30 percent every year Number of e-services and socio-economic Increase in standardization and improvement in information available through e-Gateway (MIC), transparency and administrative processes portal sites (GSO, Hanoi and Danang) (N) Significantly lowered transaction costs for Number of G2B services available online (R) targeted government services (G2C and G2B) Increase the number of private firms using ICT and e-services in Danang (R) Increase in user satisfaction for e-government services and visits to City Portal Subproject 1: ICT Leadership and MPT ICT Leadership and MIC Modernization (MIC) Modernization (MPT) Workflows and reporting lines streamlined E-government framework and common IT Design and implementation of e-government through hardware upgrades and process standards issued for all government agencies framework, common ICT standards and ICT policies automation ICT policy and implementation capacity Completion of ICT infrastructure upgrade to enhance development in at least two major areas communications between MIC and DICs, and MPT and DPT IT architecture in place, including Improved ICT awareness and management MOFA (R) flagship portal allowing interaction and capacity through training programs Completion of 3 licensing e-services and the visa transactions between MPT and government, online service offered to businesses, citizens and business and citizens Improved MOFA e-visa services under MIC foreigners available online (C) 50 oversight At least 7 appropriate services made online by Time savings of new systems (3 licensing e-services 2010 through the flagship portal Enhanced MIC and DICs communications and visa online service) (N) through ICT infrastructure upgrade Reduced transaction costs (# days for response Improved ICT awareness and management capacity turnaround) on average for the 7 e-services Core MIC services for quality and security through training programs (C) offered through the flagship portal. assurance Quality and security assurances for ICT services in IOs and CIOs from various government agencies 3 licensing e-services offered to businesses Vietnam improved through core MIC services (R) and organizations trained in ICT issues relevant and citizens available online to their position and responsibilities (technical, 200 CIOs/IOs trained legal, knowledge-sharing) 200 MIC staff trained in policy formulation, A CIO forum established to engage CIOs and capacity-building and implementation MPT staff in the next IT Master Plan (2006-11) guidelines for IT and telecommunications and general ICT Roadmap sectors MPT officials trained in ICT policy and regulatory issues and are knowledgeable about best practices in the sector Selected DPT staff trained in ICT proficiency and usage Improved security around e-services through establishing CERT, CAs and public key infrastructure that are inter-operable and in line with international standards Improved Monitoring and Evaluation of the National ICT Strategy E-government framework and IT standards in place to guide nationwide e-government applications and services GSO Modernization GSO IT architecture upgrade in place, database GSO IT architecture in place Design and implementation of GSO IT architecture 51 integration and inter-operability completed (R) Improved data transfer through LAN GSO and 50 percent of the provincial statistical connection between GSO, 3 IT Centers and Completion of ICT infrastructure upgrade (improved offices connected through LAN streamlining data data transfer through LAN connection between GSO, dissemination Provincial Statistical Offices (PSO) 3 IT Centers, and Provincial Statistical Offices (R) Shorter data collection and dissemination cycles Increased efficiency of data management and Time saving of new data management and report report generation through database integration generation through database integration (R) Enhanced efficiency of dataflow and quality of social, economic and poverty-related data Shortened survey data collection and Time saving of survey data collection and processing processing through new system (data scanning, capture, and Online access to data collection methods, some processing) (R) data analysis and some raw data Improved transparency through wider access to data and shortened turn-around time for Number of staff trained to increase knowledge on Data demands of government agencies, research data requests through GSP Portal ICT issues relevant to further modernization of GSO institutions and other data users met in a shorter functions (R) period of time Improved capacity of staff in performing streamlined and ICT-enabled data processes Number of survey databases available through GSP Portal (R) Subprojects 3, 4 and 5: E-applications in Hanoi, Danang and Ho Chi Minh City Hanoi and Danang City-level government IT architecture in place, City-level government IT architecture in place Hanoi: Design and implementation of city-level including front and back-end solutions enabling (Hanoi, Danang) government IT architecture (C) online transactions Better access to government services through Danang: Design and implementation of city-level IT Selected e-applications online for both G2G and the City Portal (Hanoi and Danang) architecture G2B and increased user satisfaction Improved ICT awareness and management Hanoi: Number of public services available through Reduced transaction time and cost through capacity of city officials through training City Portal (R) modernization of govt offices and providing programs (Hanoi and Danang) online services Danang: Better access to government services Improved transparency and reduced through different channels (R) Increased ICT use by SMEs transaction cost (i.e., time) for public procurement (Danang) Hanoi: Reduction of time required for one Enhanced competitiveness of SMEs through transaction of selected government services through training and IT services provided at the Improved transparency and reduced City Portal compared to traditional way (R) transaction center transaction costs (i.e., time) for business 52 registration (Danang) Hanoi: Number of city officials take CIO training Government officials and ICT staff in the city courses (R) government trained in e-government and IT Improved transparency and reduced issues transaction costs (i.e., time) for vehicle Danang: Improved transparency and reduced registration and driver’s license (Danang) transaction costs (i.e., time) for business registration and investment licenses (R) Improved awareness and use of ICT among private enterprises, including SMEs (Hanoi, Danang: Improved transparency and reduced Danang) transaction costs (i.e., time) for driving license (R) Danang: Number of participants from SMEs take IT training to enhance IT skills for their business activities in e-commerce, e-procurement and e- services (R) Danang: Improved ICT infrastructure of the city govt (N) Danang: Improved transparency and reduced transaction costs (i.e., time) for Construction Permits (N) Danang: Enhance and develop ICT human resources in Danang City (N) Danang: Enhance efficiency of public transport management (N) Danang: Enhance the efficiency of management of water resources quality (N) Danang: More efficient government administration (N) 53 D. Status of Procurement Packages, all Subprojects (end-2009 and end-2013) Package Cost Proc. Status at end-2009 Status at end-2013 USD Method Planned Actual Planned Actual Danang City 1. DNG 1: Vehicle 44,976.00 Shopping Contract completed Contract completed 2. DNG 5a: Design, supply, 4,700,536.93 ICB Issuance of BD BD Preparation Completion of Completion of installation, testing and Contract Contract commissioning of Danang Implementation Implementation Metropolitan Area Network 3. DNG 5b: Supply & 27,275.25 ICB Issuance of BD BD Preparation Completion of Completion of Installation of End-User Contract Contract Computing Infrastructure for Implementation Implementation the Departments and Agencies of Danang City 4. DNG 6+7: Development and 49,021.67 ICB BD Preparation BD Preparation Completion of Completion of Deployment of ICT Contract Contract Applications for Danang Implementation Implementation Authority Units 5. DNG 8a: Design, Supply, 1,269,457.32 ICB Issuance of BD BD Preparation Completion of Completion of Installation, Testing and Contract Contract Commissioning of Danang ICT Implementation Implementation Transaction Center 6. DNG 8b: Design, Supply, 1,434,406.87 ICB BD Preparation BD Preparation Completion of Completion of Installation, Testing and Contract Contract Commissioning of Danang Data Implementation Implementation Center 7. DNG12: Establishment of 1,189,605.09 ICB Not available Not available Completion of Completion of Public Wifi System for Danang Contract Contract City Implementation Implementation 8. DNG13: Development of 1,119,610.78 ICB Not available Not available Completion of Completion of Additional Public Services for Contract Contract Danang City Implementation Implementation 8. DN15:Supply and 3,223,857.43 ICB NA NA Completion of Completion of Installation of Additional IT Contract Contract 54 equipment and maintenance Implementation Implementation tools 9. DNG 0: Hire procurement 29,676.58 International Completion of Completion of specialist Consultant Contract Contract Implementation Implementation 10. DNG 2: Project 57,417.56 QCBS Completion of Completion of Management Consultant Contract Contract Implementation Implementation 11. DNG 3: Access key ICT 177,593.32 QCBS Completion of Completion of infrastructure & applications Contract Contract and identity priority business Implementation Implementation processes/s services 12. DNG 4: Develop strategy, 177,643.07 QCBS Completion of Completion of roadmap, architecture, standard, Contract Contract and prototype of e-government Implementation Implementation of Danang City 13. DNG 9: Design ICT HR 154,079.18 QCBS Completion of Contract Completion of Completion of development master plan and Contract Implementation Contract Contract high-level training program Implementation Implementation Implementation 14. DNG 11a: Hire local 29,160.60 QCBS Completion of Completion of consultant to give instruction Contract Contract and support in the project Implementation Implementation 15. DNG 11b: Recruit local 38,422.94 QCBS Completion of Completion of consultants to work for Danang Contract Contract PIU Implementation Implementation 16. DNG 14: Assessing the 436,565.10 QCBS NA NA Completion of Completion of Status of Danang E- Contract Contract Government Implementation Implementation Implementation and Building its e-Government Model Based on the Assessment 17. Purchasing Pick-up Truck 35,385.26 Shopping NA NA Completion of Completion of to serve for maintenance of Contract Contract Danang MAN and Public Wi-fi Implementation Implementation 18. Purchasing Equipment for 65,263.35 Shopping NA NA Completion of Completion of IC Design Contract Contract Implementation Implementation 55 Training and Workshop Plan 19. CIO Training program 12,319.33 Shopping Completion of Completion of Contract Contract Implementation Implementation 20. IC Design Training Course 74,011.88 Shopping NA NA Completion of Completion of Contract Contract Implementation Implementation 21. Enhance skills on 72,495.80 Shopping NA NA Completion of Completion of development of open source Contract Contract software Implementation Implementation 22. Implementation of The 445,000.00 Shopping NA NA Completion of Completion of Project Training Programs Contract Contract Implementation Implementation Seminars and Study Tours 23. Launch Workshop 1,306.48 Shopping Completion of Completion of Contract Contract Implementation Implementation 24. Study Tour 109,526.43 Shopping Contract Contract Completion of Completion of Implementation Implementation Contract Contract Implementation Implementation Table 2: MIC PIU 25. MPT 0: Procurement 100,000.00 IC Completion of Completion of Completion of Completion of Consultant (Mr. Pham Tuan Contract Contract Contract Contract Phan, Nationality: Vietnamese) Implementation Implementation Implementation Implementation 26. MPT0.1: Procurement 30,000.00 IC Completion of Completion of Completion of Completion of Consultant Contract Contract Contract Contract Implementation Implementation Implementation Implementation 27. MPT0.2: Technical Advisor 100,000.00 IC Completion of Completion of Completion of Completion of for Procurement Contract Contract Contract Contract Implementation Implementation Implementation Implementation 28. MPT 1: Vehicle Mitsubishi 50,000.00 Shopping Completion of Completion of Completion of Completion of Pajero Contract Contract Contract Contract Implementation Implementation Implementation Implementation 29. MPT 2: Project 1,200,000.00 QCBS Start of Contract Completion of Completion of Completion of Management Consultant Implementation Contract Contract Contract Implementation Implementation Implementation 56 30. MPT 2.1: Additional Project 400,000.00 QCBS Completion of Management Consultant Contract Implementation 32. MPT 12: Consultancy on 50,000.00 CQS Completion of Completion of Completion of Completion of Study, Analysis and Draft of e- Contract Contract Contract Contract Government Interoperability Implementation Implementation Implementation Implementation Framework 32. MPT 13: Amendment of the 50,000.00 CQS Completion of Completion of Completion of Completion of Feasibility Study Contract Contract Contract Contract Implementation Implementation Implementation Implementation 33. MIC 1-1: Technical advice 500,000.00 QCBS Start of Start of Completion of Completion of on designing a PKI scheme to Procurement Procurement Contract Contract support e-government Implementation Implementation development and national e- authentication framework 34. MIC 1-2: Implement the 2,700,000.00 ICB Issuance of Start of Completion of Completion of upgrade of ROOT CA Procurement Procurement Contract Contract Documents Process Implementation Implementation 35. MIC 1-4: Technical advice 1,300,000.00 QCBS Issuance of Completion of Completion of on e-Government Framework Procurement Contract Contract and IT Applications Standards Documents Implementation Implementation 36. MIC 1-5: Technical advice NA QCBS Issuance of Cancel Cancel Cancel on solution and implementation Procurement plan for e-document Documents interchange system 37. MIC 1-6: Technical advice 325,000.00 QCBS Issuance of Completion of Completion of on solution and implementation Procurement Contract Contract plan for integrated email system Documents Implementation Implementation 38. MPT 3: Consultancy on 500,000.00 QCBS Issuance of Cancel Cancel Cancel Review and Assessment of MIC Procurement and DIC’s existing ICT Documents infrastructure, future technology needs in public administration and deployment of e-Government public services 39. MIC 2-1: Issuance of 4,250,000.00 ICB NA NA Completion of Completion of 57 Procurement Documents Contract Contract Implementation Implementation 40. MIC 2-2: Completion of 4,000,000.00 ICB NA NA Completion of Completion of Contract Implementation Contract Contract Implementation Implementation 41. MIC 2-3: e-Gateway for 220,000.00 ICB NA NA Completion of Completion of MIC (Phase 2 Portal) Contract Contract Implementation Implementation 42. MIC 2-4: Implementation of 500,000.00 ICB NA NA Completion of Completion of integrated information system Contract Contract & three public services Implementation Implementation 43. MIC 2-5: Installation of 3,000,000.00 ICB Start Procurement Start Procurement Completion of Completion of Intelligent System and Unified process process Contract Contract Communication System Implementation Implementation 44. MIC 3-1: Consultancy on IT NA QCBS Start Procurement Start Procurement Cancel Cancel sector monitoring and process process assessment 45. MIC 3-3: Consultancy on 350,000.00 QCBS Start Procurement Start Procurement Completion of Completion of Policy formulation, Capacity process process Contract Contract Building and Implementation Implementation Implementation Guidelines on IT 46. MOFA 1 (MIC 4-1): Data 3,669,200.00 ICB Start Procurement Start Procurement Completion of Completion of center process process Contract Contract Implementation Implementation 47. MOFA 2 (MIC 4-2): E-Visa 1,117,300.00 ICB Start Procurement Start Procurement Completion of Completion of Application process process Contract Contract Implementation Implementation Table 3: MIC PCU 48. PCU 1: Vehicle: Mitsubishi 50,000.00 Shopping Completion of Completion of Completion of Completion of Pajero,Vinh Quang Ltd., Co. Contract Contract Contract Contract Nationality: Vietnamese Implementation Implementation Implementation Implementation 49. PCU 2: Equipment for PCU 120,000.00 NCB Completion of Completion of Completion of Completion of to manage and monitoring the Contract Contract Contract Contract project Implementation Implementation Implementation Implementation 50. PCU 2.1: Completion of 30,000.00 Shopping NA NA Completion of Completion of Contract Implementation Contract Contract 58 Implementation Implementation 51. PCU 2.2: Net meeting 30,000.00 Shopping NA NA Completion of Completion of software to support the Contract Contract communication between WB Implementation Implementation and PIUs 52. PCU 7: Upgrade Financial 25,000.00 Shopping Completion of Completion of Completion of Completion of Management Software Contract Contract Contract Contract Implementation Implementation Implementation Implementation 53. PCU 7.1: Completion of 7,000.00 Shopping NA NA Completion of Completion of Contract Implementation Contract Contract Implementation Implementation 54. PCU 9: Upgrade LAN for 10,000.00 Shopping Completion of Completion of Completion of Completion of PMU-MIC Contract Contract Contract Contract Implementation Implementation Implementation Implementation 55. PCU 0: Procurement 100,000.00 IC Completion of Completion of Completion of Completion of Consultant (Mr. Alexandru Contract Contract Contract Contract Aurelian Stoea. Nationality: Implementation Implementation Implementation Implementation Rumania) 56. PCU 0.3 (PCU 0.1 & PCU 50,000.00 Cancelled 0.2): Technical Advisor for MOFA project 57. PCU 4: Portal for the 50,000.00 CQS Completion of Completion of Completion of Completion of Project (ICT Service Contract Contract Contract Contract Engineering JSC) Implementation Implementation Implementation Implementation 58. PCU 5: Consultancy for 100,000.00 CQS NA NA Completion of Completion of monitoring and evaluation of Contract Contract the quality and effectiveness of Implementation Implementation project implementation 59. PCU 6: Auditing 42,000.00 CQS Start of Contract Start of Contract Completion of Completion of Contract Contract Implementation Implementation 60. PCU 6.1: Auditing to the 50,000.00 CQS NA NA Completion of Completion of end of Project Contract Contract Implementation Implementation 61. PCU 6.2: Additional 17,000.00 CQS NA NA Completion of Completion of Auditing for Project Extension Contract Contract period Implementation Implementation 59 62. PCU 8: Financial consultant 20,000.00 Cancelled to support in closing project 63. PCU 10: Media Campaign 150,000.00 CQS NA Completion of Completion of for VN ICT project NA Contract Contract Implementation Implementation Table 4: Hanoi PIU 64. HAN 1: Vehicle (car for 50,000.00 Shopping NA Completed 2007 project) 65. HAN5: Development 3,500,000.00 ICB Submit Approval Completed Completed March Hanoi Data Center Procurement Plan Procurement Plan 2013 by PPC 66. HAN6: Upgrade City Portal 1,200,000.00 ICB Submit Approval Completed Completed Dec. and gateway Procurement Procurement Plan 2013 by PPC 67. HAN 7: Implementation of 895,000.00 ICB Submit Approval Cancel Cancel e-document workflow Procurement Procurement Plan by PPC 68. HAN0: Hire Procurement 100,000.00 IC Completed Completed 2008 Consultant to support Hanoi PIU in 1st phase 69. HAN0a: High Level 25,000.00 IC Award contract Completed Procurement Consultant: Select a local consultant with rich experience in management and procurement for ODA project, especially WB funded projects. 70. HAN0b: Technical 55,000.00 IC Award contract Request NO for Completed Completed 2010 Consultant TOR 71. HAN0c: Individual 50,000.00 IC Submit Submit TOR to Cancel Cancel technical Supervision Procurement WB for review Consultant 72. HAN3: Access Key ICT 490,000.00 QCBS Submit Implementation of Completed Completed 2010 Infrastructure Applications procurement contract (Leading Associates, USA) 73. HAN4: International 500,000.00 QCBS Select Consultant Evaluate Proposals Completed Completed Dec consultancy to develop strategy, 2013 60 roadmap and IT Architecture and standards for new 10-year Plan for Hanoi City 74. HAN9: Design ICT HR 410,000.00 QCBS Select Consultant Submit Plan Cancel Cancel Development Master Plan and High-Level Training Program 75. HAN10: Contract 30,000.00 ICB Complete Contract Implementation of Completed Completed 2011 Administrative Legal Expert to Contract support Hanoi PIU in legal issues of project implementation. 76. HAN 11: Local Technical 30,000.00 ICB Submit Approval of Cancel Cancel Consultant Procurement Procurement Plan PPC GSO PIU 77. GSO 1: Vehicle 47,355.00 Shopping Contract completed Contract completed 78. GSO 5-2: 4,851, 916.00 ICB Contract completed Contract completed GSO infrastructure upgrade; Integrated Statistical Information System; Intranet Portal 79. GSO 5-2a: Information Warehouse and Information Dissemination Portal 80. GSO 6-1b: Equipment and 1,569,961.00 ICB BD Preparation BD Preparation Contract completed Contract completed Software Upgrade (HIPT Group) 81. GSO 6-1b: Data Collection 4,140,898.00 ICB Start contract Start contract Complete contract Contract completed and Processing solution for execution execution GSO 82. GSO 8.1: Equipment 29,843.00 Shopping Completion of Contract completed support for GSO 6-1b and 2009 contract Census 83. GSO 8-2a: Tools support 23,292.00 Shopping Completion of Contract completed for GSO 6-1b and 2009 Census contract – Cutting Machines 61 84. GSO 8-2b: Tools support 18,484.00 Shopping Completion of Contract completed for GSO 6-1b and 2009 Census contract – Air Conditioners 85. GSO 8-3: GSO Application 29,800.00 Shopping Completion of Contract completed Support Mapping Contract Administration Field Database in Vietnam: Software Engineering and GIS Center 86. GSO 8-4: Rack cabinets for 26,418.00 Shopping na na Contract Contract completed PSOs; Supporting equipment completion from SSIC 87. GSO -0: Hire Procurement 142,137.00 ICB Completion of Contract completed Specialist contract 88. GSO 2-1: Local Project 10,800.00 ICB Completion of Contract completed Management Consultant contract 89. GSO 2-2: Local Financial 17,305.00 ICB Completion of Contract completed Management Consultant Contract 90. GSO 2-3: Technical 276,395.00 ICB Bid Preparation Completion of Contract completed Advisor Consultant contract 91. GSO 3-2: Local Training 1,839.00 ICB na na Completion of Contract completed Consultant – Specialist contract (Individual) 92. GSO 3-3: Local Project 8,867.00 ICB na na Completion of Contract completed Manager contract 93. GSO 5-1: ICT Assessment 760,707.00 QCBS Contract completed Network, IT Architecture and Security Consultancy for GSO and VPN Pilot 94. GSO 6-1a: Enterprise 1,583,118.00 QCBS Bid Preparation Completion of Contract completed Architecture and Meta-Data contract Study; and IT/EA Governance Support 62 Annex 3. Economic and Financial Analysis The following analyzes the impact of the World Bank Information and Communication Technology (ICT) Project in Vietnam. It looks at (1) efficiency gains of specific interventions as well as (2) overall project efficiency. This includes the Danang subproject and General Statistics Office (GSO) subprojects. In many instances the precise impact is difficult if not impossible to measure due to a lack of baseline data and detailed financial and operating information. Instead, the analysis reports what can be reasonably assumed based on documented experiences elsewhere. Efficiency gains of specific interventions Project interventions have impacts on three groups: government, businesses and citizens. For government, efficiencies from ICT include higher output with lower expenses due to cost savings including reduction in duplication and service delivery times. Citizens and businesses achieve greater efficiency through time and cost savings when accessing government information and services. There are also benefits that are difficult to measure including the contribution of online public services to the quality of life of businesses and citizens; improved business and citizen decision making due to the availability of easier to access information; and linkages between government and private sector to create new business opportunities (e.g., open data). Danang Subproject: The Danang Subproject had impacts on government, Small and Medium Enterprises (SMEs) and citizens. Government According to the Vietnam project teams, interventions impacted the government positively due to the transition from manual to computerized methods: • The competence of city officials grew to a higher level due to knowledge and skills they received through intensive IT training. This opened the way for solid and sustainable development of the city in the coming years. • A citizen-centric government has been established where interactions between individuals, businesses and government departments will become more streamlined and easier. • Coordination and interconnection among departments and agencies in delivery of public services improved. The transition from manual to computerized processes results in lower costs and shorter processing times. Measuring this requires an assessment of the amount of time spent and costs carrying out processes before and after the intervention. In the absence of such information, some inferences can be made: • Government ICT competency was increased through training programs leading to understanding of how to implement IT systems with consequent efficiency gains. Further awareness of IT benefits is likely to reduce resistance to restructuring business processes. • Interactions between government, businesses and citizens became more streamlined resulting in quicker processing times. 63 • Coordination among different public agencies resulted in a reduction of duplication leading to greater efficiency and cost savings. For example multiple databases maintained by different city departments were streamlined with common codes and system architecture, reducing duplication and simplifying maintenance. • It is also noteworthy that Danang has ranked in the top position of the Vietnam ICT Index for five straight years (2009-2013) a period that coincides with the ICT Development Project. Note that prior to the intervention a stakeholder consultation in 2004 rated the city’s status of ICT development as limited/very limited. 23 SMEs Several interventions impacted SMEs including: • Application systems were built to meet SME requirements including the “Business Registration System", "Investment Project Management System", "Fishing Boat Management System", and "Construction Permit Management System"; • Significant information was provided for SME business activities via the Contact Center (76% public services related to SMEs); • 400 courses in ICT are available in e-Learning Portal; • 75 software developers from SMEs have gone through Software Engineering training courses; • 50 IT experts from the private sector joined the training course about the Danang e- Government Platform. There is evidence that these types of interventions are likely to have positively impacted SMEs in Vietnam. For example, World Bank research suggests that development of online business registration systems that reduce complexity, time and costs lead to a rise in business registrations and employment as well as generating income for the government. 24 Similarly, facilitating construction permits through online applications saves businesses time and money. 25 The contact centers would facilitate business activities by providing a medium for enterprises to easily contact government agencies about matters such as policy, operating hours, general information or to voice a concern. This reduces costs for both the government and business. The training interventions will likely have increased the ability of SMEs to use relevant e- government services as well as incorporate ICTs in their businesses raising productivity. 23 World Bank. 2014. "Vietnam - ICT Development." Report No: ICR 2976 24 World Bank. 2013. Doing Business 2014: Understanding Regulations for Small and Medium-Size Enterprises. Washington, DC: World Bank Group. http://www.doingbusiness.org/reports/global- reports/doing-business-2014 25 Op. cit. 64 Citizens A number of interventions were aimed at citizens: • Over 315 public services delivered via the Danang Portal for citizens; • Basic IT training courses (68 sessions) delivered on television; • 400 courses in ICT are available on the e-Learning Portal; • 1,125 citizens received IT training via the Danang Training and Applied Research Center; and, • Danang Public Wi-Fi with 329 Access Points provides Wi-Fi coverage over the city center including all main city routes, beaches, offices of departments and agencies, communes and wards and public areas to serve citizens and tourists. The delivery of online public services benefits citizens by reducing the cost and time of travelling to and waiting at, government offices. Sri Lanka showed significant savings in time from implementing online citizen services: the period to issue birth, marriage and death certificates dropped from hours to just a few minutes.26 A survey of European citizens found that 80% replied that online public services saved them time, 76% appreciated the flexibility of conducting government processes anytime and from home and 62% replied they saved money. 27 Similarly a survey carried out for Danang found that 45% of respondents expected the project’s e- services to save time, 35% stated it would be more convenient, and 23% expected to save costs. 28 Assuming that the Internet penetration rate is the same in Danang as the nationwide average, then around a third of citizens could take advantage of the online public services implemented through the project. Availability of different ICT training options via television, online portal and physical classes promotes digital literacy of citizens, providing flexibility in how they can access learning opportunities. This will enable them to more successfully use e-government as well as other online services and applications. City-wide Wi-Fi availability saves citizens the time of having to find a physical location when they need use the Internet. Since the intervention was free, it also saves citizens money. Note that there are also benefits to government in terms of making online public services more accessible and allowing government employees to access the Internet when outside the office. Businesses also benefit; for example customers could find businesses easier using location services over the Internet and business could exploit social media and other applications to create innovative relationships with customers. 29 26 Information and Communication Technology Agency of Sri Lanka (ICTA). 2011. ICT Usage in Government Organizations Survey. 27 European Commission. 2013. “eGovernment Improving but Citizens Ask for More.” Press Release, May 28. http://europa.eu/rapid/press-release_IP-13-466_en.htm. 28 An Vien Media. 2013. “Report on Performance Assessment of the Communication Campaign for Vietnam ICT Development Project 2013.” 29 According to one source, other impacts of the free city Wi-Fi include "...better IT infrastructure; boost trade promotion, lure more investment, advertise tourism images and raise cultural and educational 65 Estimated costs savings to citizens: An estimate of the cost savings to citizens from free Wi-Fi can be made by calculating the cost of Internet access for which they no longer have to pay. This makes the following assumptions: • The proportion of Internet users in Danang is the same as that across the country (36% in 2012). 30 This results in an estimate of around 345,700 Internet users in Danang. 31 • It assumes that one third of such Internet users would use the free citywide Wi-Fi service. 32 • The cost of a mobile data package is VND 70,000 33 (US$3.36 34) per month. Based on these assumptions the savings to citizens would be US$383,409 a month (114,081 users X US$3.36 (cost of Internet access per month)). The projects also supported quality of life interventions for citizens such as e-service for Water Quality Monitoring (financed for Danang) and improved bus/transport services (GIS equipped/Danang). GSO Subproject: For the GSO subproject, the following efficiency gains were observed:  Data collection and processing for the 2009 Census was reduced to 14 months from 24 months (42% reduction) as a result of project-supported ICT improvements. The calculated cost savings is US$500,000 based on the 450 data entry staff used for the 2009 Population Census 35 and assuming they receive the average monthly income for standards for the city." See: “Da Nang Provides Free Wi-Fi Coverage.” 2012. VietNamNet, May 11. http://english.vietnamnet.vn/fms/travel/51876/da-nang-provides-free-wi-fi-coverage.html. 30 VNNIC. 2012. Report on Vietnam Internet Resources. http://vnnic.vn/sites/default/files/tailieu/ReportOnVietNamInternetResources2012.pdf 31 Based on the 2012 Da Nang population of 973,800. See: "Area, population and population density in 2012 by province" on the GSO website at: http://www.gso.gov.vn/default_en.aspx?tabid=467&idmid=3&ItemID=14459 32 Based on a study of Thai Internet users which found that 35.3% used the country's free Wi-Fi service. See: ETDA. 2013. Thailand Internet User Profile. 33 For 600 MB of data. See the Mobifone website at: http://www.mobifone.com.vn/portal/en/services/home/internet/dichvu3g/fast_connect_gc.jsp 34 Converted to US$ using the 2012 annual average exchange rate. See: http://data.worldbank.org/indicator/PA.NUS.FCRF 35 GSO. 2007. Plan for the 2009 Population and Housing Census of Vietnam. 66 government workers. 36 The difference in salary costs between data entry staff working 24 months as compared to 14 months is US$500,276.  Forms scanning hardware used for the 2011 Agriculture and Rural Census generated a 50% improvement in overall processing time with time reduced from 2 years to 1.  The time for data collection and processing for Consumer Price Index (CPI) surveys was reduced by one third from 15 to 10 days following the introduction of new IT processes. In addition, the GSO subproject reduced duplication significantly by harmonizing and integrating databases into a standard system architecture. The provision of additional free statistics online would also have improved decision making by both the government and the private sector. 37 Overall project efficiency Several factors contributed to overall project implementation efficiency for Danang and the GSO. Two key factors were: (i) the skills of the PIU; and (ii) competitive bidding. The Danang PIU project manager had significant expertise in both technology and project management. Further he had good relations with local leaders at all levels. This helped speed up the local authority’s approval for all submissions. Similarly, there was good coordination between the GSO PIU and technical and business staff, enhancing work flows and reducing implementation time. The competitive bidding process for all Danang packages resulted in significant cost savings. The original overall budget of the Danang Subproject was US$ 17,310,000. In 2010 when the project was restructured, additional funding of US$10,000,000 was allocated due to successful implementation. This brought the total value of Danang Subproject to US$27,310,000. Following 5 years of implementation, Danang PIU has saved a total of US$6,146,000 through competitive procurement practices. This allowed the PIU to acquire additional equipment with the savings. Though not as large as Danang, the other PIUs also achieved cost savings on procurement packages. In aggregate, project savings totaled over US$15.0 million and most of this is likely to have been used for other equipment, systems etc. Conclusions Although detailed information necessary for carrying out a comprehensive efficiency analysis is lacking, the anecdotal data suggests that the project interventions are already having an impact on reducing costs for government and saving time for SMEs and citizens. These benefits can be expected to grow as the e-services evolve in scope and functionality and more Vietnamese use the Internet. 36 The average monthly income for government workers was Dong 2.3 million (US$111) in 2011. See "Monthly average income per employee in state sector at current prices by kind of economic activity" on the GSO website at: http://www.gso.gov.vn/default_en.aspx?tabid=467&idmid=3&ItemID=14384 37 For example the Finnish Ministry of Employment and the Economy found that putting statistical data online made it easier for entrepreneurs to search for data and improves analysis. The system gets about 3,000 queries a month. See: Tieto, "Customer case: The Ministry of Employment and the Economy" at: http://www.tieto.com/sites/default/files/atoms/files/case_ministerio_en_1601.pdf 67 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Christine Zhen-Wei Qiang Task Team Leader CITPO Natasha Beschorner Senior ICT Policy Specialist CITPO Kashmira Daruwalla Senior Procurement Specialist CITPO Naomi J. Halewood ICT Policy Specialist CITPO Andrea Ruiz-Esparza Senior Program Assistant CITPO Peter L. Smith Consultant CITPO Supervision/ICR Lead E-government Specialist and Deepak Bhatia TWICT TTL Kaoru Kimura ICT Policy Specialist TWICT Christine Zhenwei Quiang Task Manager CICTI Dung Thi Thuy Dao Program Assistant EACVF Kashmira Daruwalla Senior Procurement Specialist ECSO2 CTRDM Quyen Do Duong Finance Analyst - HIS Naomi J. Halewood ICT Policy Specialist TWICT Hung Viet Le Sr. Financial Management Specst EASOS Hanh Thi Huu Nguyen Financial Management Specialist EASFM Samia Melhem Lead ICT Policy Specialist TWICT Hoang Xuan Nguyen Procurement Specialist EASR2 Son Duy Nguyen Senior Operations Officer EACVF Dzung The Nguyen Senior Rural Development Specst EASVS Tam Thi Do Team Assistant EACVF Rohit Gawri Information Analyst SARIM OPCIL- Hoi-Chan Nguyen Consultant HIS Cung Van Pham Sr Financial Management Specst EASFM Quynh Xuan Thi Phan Financial Officer GEFOB Rajesh B. Pradhan Consultant FIEID Andrea Ruiz-Esparza Senior Program Assistant TWICT Randeep Sudan Sector Manager TWICT Kien Trung Tran Senior Procurement Specialist EASR2 Anh Tran Phuong Vu Administrative Officer EAPC2 Huong Thi Thu Vu Program Assistant EACVF Anna Roumani Consultant TWICT 68 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD ‘000 (including travel No. of staff weeks and consultant costs) Lending FY03 3.33 68.25 FY04 25.46 112.02 FY05 41.05 181.56 FY06 10.41 46.23 Total: 80.25 408.06 Supervision/ICR FY06 21.61 75.21 FY07 19.03 82.85 FY08 13.91 75.64 FY09 30.49 131.53 FY10 23.68 117.96 FY11 16.54 121.62 FY12 38.40 201.77 FY13 42.58 182.34 FY14 17.18 76.52 Total: 223.42 1,065.44 69 Annex 5. Beneficiary Survey Results 5.1 The following summarizes the results of a major study and an important assessment. The Project financed a study in 2013 under procurement contract DNG14 entitled: “Assessing the Status of Danang e-Government Implementation”, conducted by consultants Leading Associates and KDi Asia. It is important to note that the study does not focus exclusively on the ICT Development Project but it does refer to specific results from the Project where relevant. The study is important for providing a roadmap of how evaluation of ICT projects in Vietnam might be conducted from here on, establishes a form of baseline for doing that, and emphasizes building strong M&E of ICT programs. Its findings – specifically those pertaining directly to the Bank- supported project - are summarized below along with important lessons for such projects. The second report/assessment is entitled “Report on Performance Assessment of Communications Campaign for Vietnam ICT Development Project” conducted by An Vien Telecommunications and Communications Company in December, 2013 under contract to the MIC PCU. Danang Study: 5.2 Methodology: The study proposes and itself uses, the e-Government Assessment Framework (eGAF), which provides a comprehensive review of e-government from three important domains: ICT Outcomes which assesses user satisfaction, service quality, information quality and systems quality; ICT Program Management assesses quantitative data from diverse sources for six dimensions: projects, applications, infrastructure, security and architecture, regulations and human resources; and ICT Governance which looks at five dimensions via questionnaires and an interview-sourced capability maturity assessment in each selected public agency. Conducted under the DNG 14 procurement package, Leading Associates/KDi Asia interviewed 75 SMEs, 642 citizens and 38 public sector departments and agencies - the latter including health, transport and science and technology agencies - and Peoples’ Committees. A summary of the survey framework is attached but does not contain the citizen or business questionnaires (requested from firm but not provided). Results of e-GAF: ICT Outcomes Assessment: This domain used qualitative, survey/questionnaire-sourced data from citizens, businesses and public agencies: • E-Government in Danang City has brought the city to the top position of the Vietnam ICT Index for five years continuously (2009-2013). • This occurred within the life of the ICT Development Project and in a city where a stakeholder consultation in 2004 rated the city’s status of ICT development as limited/very limited under major themes. 38 • E-Government in Danang City has entered a maturing phase of its development; 38 Ratings for Danang were as follows in 2004: (i) business process changes, ICT as a strategic tool: Very Limited; (ii) true integration, data sharing, workflow, functional applications: Limited; (iii) training of officials: some progress but low awareness and competence still a key constraint; (iv) connectivity, LANs: more required particularly at district and ward level; (v) WAN: some, but less than other cities; and (vi) basic computerization: medium progress but more needed, particularly districts and wards. 70 • The Danang ICT Transaction Center operates six days/week from 8:00 am to 4:00 pm for general enquiries about public services. • The Danang Data Center, Public Wi-Fi and Metropolitan Area Network (MAN) are providing good quality infrastructure to the public and to government agencies. • At the time of the study, online services in Danang were tending to play a greater role in providing information to the public on public service procedures than as a new method for online transactions between Government and the public. • This is explained by the still-limited ICT skills and competence of citizens and the fact that in a small city like Danang, people can still transact face-to-face with agency representatives, whose services are generally well-rated. • Citizens also believe that personal transactions foster accountability and transparency, and enable them to participate in government policy-making. • While most citizens are satisfied with public services of Danang City, the level of satisfaction for online services is expected to further increase once end-to-end services – still in the pipeline at the time of the study - are introduced. • The study recommends that the City pay attention to the digital society and knowledge economy aspects of ICT and e-Governance development, supporting people’s skills development and access to computers/literacy. ICT Program Management: This domain used quantitative data from diverse sources which with the exception of Security and Architecture, were collected from existing reports and indices (Danang ICT Steering Committee – on which the project was represented - annual ICT Current Status Report; agencies’ project planning and implementation documents; and Vietnam’s annual ICT Index): • Successful implementation of all planned procurement packages establishing key systems and deploying a comprehensive ICT infrastructure, financed by the World Bank ICT Development Project, demonstrated Danang’s commitment to strong ICT management. • Greater emphasis is needed on ICT Information Security practices. • ICT Regulations were established and will need more frequent review and updating to reflect changing needs. • An important issue is that the current human resource endowment cannot meet further ICT development/expansion needs without adequate staff and budget. • With more services migrating online and demand increasing, capacity and resources must also scale up. ICT needs and organizational framework, as well as resources must be mandated to meet sector growth. ICT Governance Assessment: This domain used questionnaires and interview-sources capability maturity assessment in each selected public agency: • Danang’s e-Government objectives are clear as enunciated in several forms such as the e- City Program which describes the modernization of the city through technology and ICT. • The city’s ICT strategy is based on consultative, inclusive processes involving stakeholders and awareness of ICT is quite high among senior city managers. • Policy instruments still do not define how ICT Governance is to be measured and the performance indicators for measuring progress. • Danang’s Enterprise Architecture (EA) was formulated in 2009 under the ICT Development Project and attention is now needed to its updating. 71 • Implementation of the Korean e-Government Standard Framework (eGovFrame) represents only the application architecture subset of the full EA scope and the EA should fully-incorporate this Framework. • Evidence suggests there is no Risk Management Framework and Monitoring and that this will need attention. • More resources are needed for managing information security oversight. • There is not yet a framework for systematic monitoring, evaluation and assessment of performance of ICT processes. While some monitoring was in place at end-project, there were no specific performance parameters for assessing the outcomes of e-Government and evaluating the effectiveness of ICT governance. • When the study was conducted, management of ICT projects was centralized in the PIU and DIC accounting for less awareness among other agencies not typically accountable for managing the ICT Development Project. • The acquisition and deployment of ICT services under the ICT Development Project have been properly established but there is need for structured, better-defined and activated processes for managing change and releasing new e-services. • More structured frameworks would improve the planning for and execution of initiatives to deliver the required level of service. • Identification and monitoring of performance standards for quality of ICT service management and governance are currently centralized, and awareness of performance metrics and targets for existing online services is still developing. Other Project-related results, events and trends reported by the study: ICT Development Project-financed procurement “packages” have had positive impacts: • Capacity of the ICT workforce was improved via the development of a Master Plan for ICT Human Resource Development (DNG9). • New, end-to-end online services based on the e-Government Standard Framework (Korea) were introduced in 2013 (DNG 6-7 and DNG13). • Danang City in late 2013 officially opened the Danang ICT Infrastructure System which includes Project-supported components: Danang Metropolitan Area Network (MAN); Public Wi-fi, ICT Training, the Applied Research Center and other related components. • The City has won several international awards from organizations such as FutureGov, the 2013 eAsia Award of AFACT, and Smarter Cities Challenges for successfully adopting ICT for citizens and to serve business organizations, as a direct outcome of the Project. • Specific examples show that the cost of public administration was decreased through the effective application of ICT systems. ICT Infrastructure Development Results: • ICT infrastructure was modernized, becoming an important contact point for the national backbone network covering the city with networks based on international standards, providing good quality, diverse services. • Danang MAN was re-constructed, connecting all government agencies, departments, districts and branches (enterprises, schools, hospitals) to support daily operations. • 100% of local government agencies are now connected to LAN and high speed Internet (ADSL) • 100% of staffs and employees working for all branches, departments and people’s committees of districts now have computers. 72 • Fiber optic network system (Metronet) with 100Mb capacity is connected to 72 government agencies. • The Danang Data Center, ICT Transaction Center, ICT Infrastructure Development Center, Public Wi-Fi network and ICT Training Center are established and operational. Human resources development: • The Human Resource endowment in Danang has improved: the ICT Development Project’s main contribution has been through the development of an ICT Human Resource Development Master Plan (package DNG 9) backed by training; • Training courses resulting from this plan included: CIO training for 27 city and department leaders; 18 training courses in e-Government and ICT governance skills for government managers and staff; technical training courses for diverse ICT stakeholders; study tours to developed countries to acquire understanding of best practices for e- Government development. • Such training has improved the quality of the ICT workforce but is hampered by ICT human resources and civil service regulations. ICT application development results: • The study asserts that the Project’s installation of One-stop Shop software in 56 communes in 2010 improved the effectiveness of government management. Managers can monitor and control administrative processes and all citizens can access the site for information. • Almost all departments and agencies have a website/portal accessible by users at all times. • Some 315 online e-services deployed to date were directly attributable to the ICT Development Project: Driving Licenses Management; Land Use Rights Management; Business Registration Management; e-Procurement; Building License Management; Geographical Information System (GIS) and others were being finalized under procurement packages DNG6-7 and DNG 13 at time of study completion. • These services are based on e-GovFrame, a Korean open-source e-Government application framework. Impact of the ICT Development Project in socio-economic development: • Strong e-governance foundation for Danang. • Positive impact on the operational capacity of the City. • Improved productivity, quality and efficiency of key public services for people and business as well as public authorities from municipal to commune level. • Significant progress towards establishing a “professional, responsible, active and effective administration for the people”. • Authorities have significantly reduced the cost of public administration through the effective application of ICT systems and improved the operating efficiency of the administration while broadening people’s participation, improved the productivity and effectiveness of city government agencies in managing and serving people, increased the transparency of government activities and reduced bureaucracy. • Equality of access to information through new channels of communication (email, text messages, phone calls/other) has improved, especially information about administrative procedures for people and business and service responsiveness has improved. 73 Other methodologies applied by the study: The e-GAF analysis is supplemented/complemented by three other methodologies seeking to correlate and measure Danang’s achievements in e-Government. The results have been used very selectively, i.e. only where the Project is specifically-mentioned to avoid attribution errors. Public Administration Performance Index (PAPI): This is a policy monitoring instrument providing objective, bottom-up, evidence-based measures of citizen’s experiences and direct interactions in regard to standards of provincial level governance, public administration and public services performance. Since 2010, PAPI has surveyed 32,500 citizens in 63 provinces, of which 13,747 in 2012. • Achievements of Danang related to the PAPI Index show close correlation to recent e- Government initiatives such as development of e-District and e-Ward systems, and fully- implemented One Stop Shop system (OSS) and Inter-Agencies One Stop Shop (IAOSS) systems. • The Danang ICT Transaction Center (including a call center), in operation since 2012, has helped make public service procedures clearer to citizens and businesses. • New online public services platform developed under packages DNG 6-7 and DNG 13 are expected to continue supporting Danang’s high rating in the PAPI Index. Provincial Competitiveness Index (PCI): This instrument assesses the ease of doing business, economic governance, and administrative reform efforts by local governments of Vietnamese provinces and cities to promote private sector development and only includes data supplied by private, domestic operations. The 2012 report surveyed the perceptions of 8,053 domestic firms. The PCI comprises nine sub-indices reflecting economic governance areas affecting private sector development. No specific attribution to the Project is intended but it is reasonable to assume that the Project made a contribution. • Danang improved its rating from 2011 to 2012 in: informal charges; developed and high- quality business support services; reduced time requirements for bureaucratic procedures and inspections. • These three components are described as having a very close relationship with the development of e-Government services in Danang City. Public Administration Reform Index (PARI): Looking at three reform indices: Index of Central Authorities; Index of Department Authority; and Index of District Authority and cross- referencing them to the initial e-GAF assessment, the study concludes that: • Danang’s e-Government initiatives and the Danang ICT Development Project in particular have had many positive impacts on the city’s socio-economic development; • Comparing the e-GAF assessment on customer satisfaction with online public services and the PAR Index on public services, Danang citizens still prefer counter to online services (explained by the ICR as also stemming from citizens’ lack of access to computers, and to still-modest IT literacy). • This explains the still modest utilization of fully-transactional online applications which in turn explains the so far modest overall impact of the e-Government program on other indices. • From 2014 onward, operation of the new ICT infrastructure financed by the ICT Development Project may represent a significant challenge to Danang and therefore a 74 “proactive ICT governance framework should be established and information security provisions should be improved”. • Importantly, to improve e-services usage rates, Danang needs a proper communications plan to promote ICT usage by stakeholders and to solicit their feedback to improve the quality of online services; Study recommendations: The study makes an extensive series of recommendations for future ICT development and programs, summarized below: ICT Monitoring and Evaluation: • Adopt the eGAF framework for monitoring and evaluating e-Government and ICT Development ensuring consistency/linkages to the City’s socio-economic goals. • Appoint a single unit accountable for coordinating and implementing assessment exercises. • Conduct the ICT Outcomes assessment to focus on satisfaction levels of citizens and businesses, twice yearly or on a continuous basis through online “pop-ups”. • Conduct the ICT program Management assessment by aggregating quantitative data on readiness from existing secondary data sources/indices, annually, supplemented by internal reports on adoption and implementation status of projects, ICT applications and infrastructure. • Conduct the ICT Governance assessment every two years using trained assessors in consultation with responsible agency personnel and provide an objective evaluation of the capability maturity of ICT management in selected public agencies. ICT Regulatory and Policy Management: • Establish a central ICT Governance entity responsible for long-term strategy which aligns ICT strategy and includes business sector; integration of city-wide technology initiatives and compliance with ICT policies and standards; and compliance management with ICT Regulations and standards. • Establish a Risk Management Framework including assessment and awareness-building, update risk registers, monitor risk mitigation initiatives, and establish risk responsibility. • Minimize overlap, redundancy and cost in ICT-related operations, ensure that proposed new systems are appropriate and a sound business investment, and improve organizational maturity and capacity in effective management of IT with clear and measurable success criteria. • Develop minimum standards for project management and oversight, define project management performance standards and methodologies, develop systems testing methodologies and promote effective portfolio management with a central pool of resources. ICT Architecture Management: • Complement Danang City’s Enterprise Architecture defined in 2009 with detailed guidelines and Reference Architecture Models to reflect 2013 ICT infrastructure, e.g., incorporate Korean eGovFrame into Danang’s reference model and rationalize same 75 within the EA’s Application Architecture. The Danang EA needs updating/revision and participation of stakeholders in this process is encouraged. • Develop a multi-year ICT Strategic Master Plan to ensure public agencies are on the same page and moving in same direction and update annually. • Develop a communications platform to disseminate information to the public and private sectors on ICT programs and initiatives. ICT Resource Management: • Establish a clear top-down framework for managing ICT services and their governance. • Danang City to establish an industry standard framework for service management with clearly-defined functions, roles, goals and expectations, based on the UK Government’s IT Service Management (ITSM) framework, and mapped to achieve specific objectives of the ICT Master Plan. • Establish a formal process to capture new e-service requests or changing operational requirements, i.e., a centralized User Representative function to interact with and interpret user needs. • With limited budgets and human resources, Danang City’s ICT agencies need to consolidate roles cleanly dividing areas requiring specific knowledge domains or authority related to a business process from generic/user/customer facing ICT processes and consolidating the latter into a singly/centralized ICT Service Unit with a Chief Information Officer. • Establish a single point of contact – an ICT organization - to manage a structured incident response process that is transparent and measurable and provides feedback and/or resolution of issues. This entity must understand the e-services and processes it supports and keep citizens informed of revisions to e-services to prevent pushback/aversion to using the particular service. This entity must maintain transparent and objective communication with all stakeholders through official channels/forums. ICT Security Management: • Establish an effective information security unit in Danang City with authority to manage all security processes across Danang’s e-services. • Formally define information security roles and functions, and monitor the implementation of security controls selected to mitigate specific risks. • Enforce security policies and standards through regular compliance reviews and audits, providing feedback to senior officials with validated risk profiles. B. Assessment of Communications Campaign for the Vietnam ICT Development Project Field interviews were conducted in December 2013 via phone interviews with samples of 300 persons per province (Hanoi and Danang), male and female in the age range 24-45. Results: • 65% of respondents in Danang read e-news daily vs. 35% in Hanoi • Respondents had mostly learned about the Project via the TV campaign or Internet (27% and 25% respectively) • About 36% of respondents were aware of and recalled the Project’s communication themes, a result indicating need for improving campaign methodology 76 • 41.7% of respondents were aware of the construction of e-government for Danang and Hanoi • 35.3% were aware of motor vehicle registration and drivers’ licensing e-services • 28.3% knew about online licenses • 21.3% knew that the Project was implemented by MIC and the World Bank • In terms of GSO modernization and electronic visa applications/processing, 12% of respondents were aware. • Benefits of the Project were seen overwhelmingly as “keeping pace with world trends“ • “Convenience” was rated higher in Hanoi (53%) than Danang (35%). Danang is a small regional city and citizens still preferred personal, face to face transactions (as noted in the Danang Study). • Relative values by type of benefit were as follows: Type of Benefit/Project Services Danang Hanoi Time savings 45% 48% Convenience 35% 53% Transparency 57% 27% Cost savings 23% 40% Simpler procedures 46% 15% Keep up with trends 46% 15% Better than old way 22% 40% • The assessment concludes that the Communications Campaign has improved the awareness level of project content. • 36% knew about and remembered the communication topic • About 10 million people are estimated to have seen/learned about the Project via TV, followed by Internet and Press. More time is needed to assess the effectiveness of the new e-services. 77 Annex 6. Stakeholder Workshop Report and Results (if any) N/A 78 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The following are Executive Summaries of the Completion Reports from MIC/PCU, GSO and Danang. A. MIC PCU: Executive Summary of Completion Report Executive Summary Overview of Vietnam ICT Development Project The Government of Vietnam (GOV) recognizes the potential of Information and Communication Technology (ICT) in its various applications to enhance competitiveness, facilitate economic diversification and increase productivity, as well as to improve the efficiency and capability of institutions, particularly in government. GOV has undertaken several initiatives to upgrade information and communications infrastructure. These include: introducing information systems into selected government departments and piloting online services; providing incentives to the ICT industry; developing ICT skills and enhancing awareness of ICT, in government and to some extent in the private sector. In 2005, the World Bank approved the project (Vietnam ICT Development Project financed by IDA Credit No 4416) for assisting Vietnam to accelerate utilization of ICTs to foster economic development, modernize public administration and improve delivery of public services to citizens and businesses. Through a combination of business process improvements and deployment of technology, the project will support a substantially enhanced government online presence and content at the national and municipal level, through interactive and dynamic portals at the MIC, and the People’s Committees of Hanoi, Danang and Ho Chi Minh City. MIC, and the above- mentioned municipal administrations would roll out e-government services to businesses in areas such as e-procurement and business/land registration. The project commenced in September 2006 and planned to complete by 30 Jun 2011, but due to slow implementation, the project was extended to 31 December 2013. Critical Factors Affecting Implementation and Outcomes Positive Conditions: (1) PMU was received the closely directs and supports from the Leaders of the Ministry of Information and Communication, specially is Deputy Minister –Dr. Nguyen Minh Hong who directly in charge of the MIC subproject in particular and overall VN-ICT Development Project in general. (2) The staffs involved in the MIC subproject (of PMU, stakeholders) are qualified, professional qualifications and enthusiasm in the work. (3) Good Project management consultant firm (MPT2) has helped to promote the progress of projects (4) Close cooperation between Stakeholders (12 DICs, MOFA and other MIC agencies) during project implementation to ensure sustainable of the project results. 79 (5) Besides that, during project implementation, MIC subproject always gets absolutely strong supports from WB. The WB task team has conducted the evaluation for every 6 months and provided the PMU-MIC the valuable recommendations to support us to solve the difficulty in procurement and financial management. Negative Conditions (1) Changing of procurement guideline and personnel from WB: In 2010, there was a change in development of IT procurement guideline together with changing of personnel in WB task team leader. Therefore, most of bidding documents had to re-submit, and some bidding documents took more than 2 years to get the NOL from WB, for example: MIC1.2, MIC1.3, MOFA1, MOFA2 (see table 3 below) (2) WB’s Comments to the IT technical documents too slow and lack of consistency in the approval of the Bank for the bidding documents (BD). (3) Changing of director of subproject in PMU-MIC: 02 times: 2008 and 2009. (4) Lack of project management experience in PMU-MIC because most of the PMU staffs were first involved in the ODA project implementation. (5) Issue with anonymous complaints: The handling of anonymous complaints was different between Vietnam and the World Bank which also caused delays in the process of implementing the packages. Achievement of PDO: Stage 1: before 2010 the achievement against the original PAD-based PDO and data/results against the original Key Performance Indicators. Event that, MIC subproject has had successfully issued the e-Government Inter-operability standard which help to standardize all IT technical standard within project and overall IT investment within government agencies. Low achievement of PDO of the subproject cause by following reasons: (1) All PIUs are new to WB IT project which having different procurement process; (2) MIC subproject has been restructuring to combine PCU and PIU into single organization PMU which to manage entire project activities. From the implementation view, MIC subproject has established 3 technical implementation working group who responsible for 3 components by 2008. Stage 2: After project restructuring the achievements against the post-Restructuring #1 PDO At the end of stage 2, most of PDO and related KPI have been archived as follows: - Unified IP communication system has been setup at MIC head quarter which to allow smoother communication within MIC. In additional 12 DIC provinces also equipped with new hardware infrastructure including Data center and video conferencing system. - MIC also support MOFA to implement MOFA data center and online visa system. By the end of 2013, online visa system is operating and new data center was utilized to consolidate existing applications of MOFA to help better IT governance within MOFA. 80 - The training and capacity building activities of MIC subproject has helped to improve the capability of MIC policy makers and also improve the IT governance capability of business sector via training to state-own enterprise and SMEs - With the support from project, several legal and policy documents have been approved including Radio Frequency Law, Postal Law, National Information Security master plan and various IT application decrees - The National PKI infrastructure also improve via implementing new root CA system which help to ensure higher securer transaction on electronic environment in Vietnam Reasons for success: - MIC subproject was meet with demand of improving infrastructure from MIC and DIC - MOFA component was exactly met the requirement from the operation of MOFA Impacts on Intended Beneficiaries (if any): - Project is helped MIC to improve the legal environment by supporting MIC to draft and approve several legal and policy documents. Those policy documents will contributing the improve IT application and overall ICT development in Vietnam in the future - The project is helped to create a comprehensive IT infrastructure at MIC and also DIC which provide a new communication way between central and provincial government - Online services, especially online VISA system helped to create better environment to the citizen and business in Vietnam Other unintended Outcomes and Impacts: - The project is help to improve the image of Vietnam to the international as as providing a faster and easier method to getting entry visa to Vietnam. It will contribute to promote the trade and tourism in Vietnam. Risk to the Project’s Development Outcomes With result and outcomes of MIC subproject, we identified following risks may effect to the sustainable of the project as follows. 81 No. Critical Risks Mitigating Activities 1 Sustainability for new • MIC should prepare the operation cost and maintenance budget to system ensure smoothly operation of newly implemented system • Provide proper training to technical staff to improve the IT system operation and management capability. • Maintain a proper IT system operation policy to increase the availability of the system 2 Handling over of new • MIC leaders and relevant stake holders should cooperate closer in system to the system post implementation period to ensure the system will be transfer operator from PMU smoothly 3 Lack of IT system • Provide frequent training course on the technology which using by management and new system operation • Continuously train the IT staffs of MIC to equip them with update technical standard 4 Resolve the cancellation • MIC should plan to mobilize the government budget to resolve the activities cancellation packages of project such as MIC2.1, MIC3.1 to setup a new data center to host the e-services and application Assessment of Bank and Client Performance 1. Project Preparation Period Project preparation at MIC has been very well implemented, but after effectiveness of project, the project plan of MIC has been changed dramatically which created very big challenges for PMU to implement and manage entire project implementation. 82 Changes in project organization structure by 2008, 2009 and 2010 also effected to the project progress at MIC. 2. Project Implementation Period - Procurement procedure of WB has been change in the midway of project implementation cause lots of challenging to PMU as the procurement and technical staffs had to learn to familiar with the new process and guideline - Very complex project organization in MIC subproject with unclear responsibility and accountability required lots of coordination activities from PMU during the project implementation. - Limited technical capability of PMU and technical working group to create various IT solutions for longer procurement cycle; some complex packages took 3 years to complete. - Even the WB team is very supportive to the project, but by the end of project, the technical advisors from WB were changed frequently and the technical advisors are less local environment understanding, therefor it created delays in approval of bidding documents. Lessons Learned 1. IT application project which usually will bring the impact to many different stake-holders and required lots of changes in the operational mind set from the stake holders. Therefore, to ensure the successful of project implementation, a detail and effective project communication and change management plan should be design and implemented throughout the project implementation. This communication and change management activities will help to reduce the resistant from the stake holders and mobilize them to participate more actively to the project. 2. To ensure greater integration of packages, ICT strategy planning should be conducted properly to identify future projects justified through strong business case. ICT as a strategic investments need to be closely aligned to the needs of the organisation’s development and the government’s vision for the service delivery of the future. The result and outcome indicators are an essential part of this planning outputs and they must originate from the formulated business strategies of the organization 3. IT application and e-service development package usually will take very long time to complete compare to hardware infrastructure packages. Therefor in eGovernment or IT application project, the project owner should consider to plan the application and e-service development package in better alignment maner with purchasing and implementing hardware infrastructure package which to ensure the application and e-services can be operate properly prior to the project closure. 83 4. The introduction of e-services and ICT applications requires a different mind-set in modern organizations i.e. a service management and service delivery improvement outlook. ICT is an integral part of government business operations and can have great socio-economic impact on the service delivery to the citizen and businesses. Making ICT a priority to support and improve services need to be on the mind of all senior government officials. ICT today is positioned to drive improvements and innovations to meet the demands of the constituents 5. The PMU staffs should equipped with proper project management and risk management prior to the project implementation. Besides that, PMU staffs also should understand the stake holder analysis and business requirement analysis in order to manage and monitor the project and vendors properly. 6. The impact and outcomes of IT project especially for e-service development project only able to assess properly after live operation of the system for at least 1 to 2 years. Therefor at this moment, its very hard to find the tangible outcomes or impact of this project at this moment. B. GSO: Executive Summary of Completion Report Overview of Vietnam ICT Development Project The Government of Vietnam (GOV) recognizes the potential of Information and Communication Technology (ICT) in its various applications to enhance competitiveness, facilitate economic diversification and increase productivity, as well as to improve the efficiency and capability of institutions, particularly in government. GOV has undertaken several initiatives to upgrade information and communications infrastructure. These include: introducing information systems into selected government departments and piloting online services; providing incentives to the ICT industry; developing ICT skills and enhancing awareness of ICT, in government and to some extent in the private sector. In 2005, the World Bank approved the project (Vietnam ICT Development Project financed by IDA Credit No 4416) for assisting Vietnam to accelerate utilization of ICTs to foster economic development, modernize public administration and improve delivery of public services to citizens and businesses. Through a combination of business process improvements and deployment of technology, the project will support a substantially enhanced government online presence and content at the national and municipal level, through interactive and dynamic portals at the MIC, and the People’s Committees of Hanoi, Danang and Ho Chi Minh City. MIC, and the above-mentioned municipal administrations would roll out e-government services to businesses in areas such as e- procurement and business/land registration. The project commenced in September 2006 and planned to complete by 30 Jun 2011, but due to slow implementation, the project was extended to 31 December 2013. 84 Coherence of the Subprojects The VNICTDP aimed to promote ICT development in Vietnam at both the national level (MIC and GSO subprojects) and the municipality level (e-Government subprojects) by illustrating: a. Leadership/systemic effects through: (a) development of ICT policies; (b) development of a common national ICT architecture and interoperability framework; (c) providing improved information/data for timely policy and decision making (GSO); (d) fostering human capacity building at the CIO level. b. Demonstration effects through: (a) effective changes in business processes; (b) scalability and sustainability of the subprojects; (c) pilot e-applications at the city level; and (d) establishment of a budget to support innovative initiatives in selected ICT areas to involve other stakeholders who are not included in the proposed project. The selection of the cities intended to cover the capital city (where e-government can be best demonstrated), a commercially-developed city (Ho Chi Minh City) and a city in a less- developed central region (Danang). Major factors affecting/influencing project implementation 1. Strategic top down project planning with mini GSO IT Master Plan at the beginning of the Project using a business driven approach helped GSO to minimize the impact of project changes throughout the Subproject. 2. Commitment of the leader and support from business department during project implementation. Many different types of beneficiaries involved make the Project become more complex and uncontrollable. 3. Government cost norms of Vietnam have a very negative impact on the procurement process as it is very hard for PIU to convince the management agencies for approval of cost estimate for specific packages. 4. The rapid changes of technology and solutions during the procurement process which was often too long (at least one or two years) led to changes required during the project implementation. 5. Changing of procurement processes by the World Bank and inconsistency of IT procurement support from WB cost the GSO Subproject almost 2 years delay from 2009 – 2011. Lessons learned 1. Good project planning from the beginning will minimize effects of changes to and risks of the Project during implementation. This shows clearly in the GSO Subproject which had only very minor changes in project scope and objectives. In order to achieve this, at the beginning of Project, GSO PIU cooperated with the procurement consultant to plan the Project to ensure close alignment between packages in the Procurement Plan and to ensure that all project activities contributed to at least one specific Project Development Objective. 2. Project must be designed on a business driven basis. All GSO Subproject activities/components have been designed to meet a specific business requirement. 85 3. Have clear plan to manage all stakeholders related to the Project, and ensure that all stakeholder are well informed about the project progress and benefits they will get as well as challenges they will face from the Project. 5. Buying the solution rather than buying equipment. It will minimize the integration issue between different packages. 6. Enterprise architecture is a powerful tool for entire project governance. Since GSO completed GSO6.1a to develop GSO EA, other related IT investment in GSO became clearer and easier to align the investment to GSO development goals. But it was a pity that the delay in selecting GSO6.1a consultant then GSO EA did not adequately support the GSO Subproject. 7. Need a stronger project monitoring and evaluation process, as lots of package implementation has been delayed. 8. GSO leader must be aware to prepare in advance the necessary resource and operational policy for new IT systems which were procured by the Project to maximize the benefits from these new IT systems. C. Danang City: Executive Summary of Completion Report Project Objectives The Subproject was designed to enhance the ICT environment in Danang; provide the basic foundation for e-government deployment in the city’s district offices and departments; develop various types of e-government services; promote e-applications for business and develop ICR HR resources. Key Design Features The Subproject consists of four key components which include: (1) Assessment, Strategy, Roadmap, IT Architecture and Standards; (2) Local e-Government Enablement; (3) E- Application, E-Commerce, and (4) ICT HR Development and Awareness-Building. The above four key components (except for e-Commerce) were successfully completed and put in operation in late 2013. Details are as follows: Component Output Assessment Strategy, Danang Enterprise Architecture has been developed as a foundation Roadmap, IT for implementing e-Government for Danang City Architecture and Standards Local e-Government An ICT Infrastructure which includes Danang MAN, Public Wifi, Enablement Data Center, Contact Center etc, has been set up and successfully put into operation in late 2013 to facilitate the delivery of public services to citizens. It consists of more than 300 public services posted on the Danang City Portal to deliver to citizens E-Application, E- 315 public services have been developed and provided to citizens and Commerce businesses via Danang Portal. E-Commerce was removed since it is part of the national system and 86 financed accordingly. ICT HR Development An ICT Human Resource Master Plan for up to 2020 was developed and Awareness- and issued in 2010. Based on this Master Plan, over 100 leaders Building joined study tours to countries with advanced e=Government models such as Singapore; Malaysia; Taiwan, China; England; Australia; and Korea. 2,000 officials and 150 IT experts from SMEs went through intensive IT training. Major Factors Affecting/influencing project implementation: 1. The commitment of the city leaders, the determination, responsibility and dedication of all members of Danang PIU, the valuable support of the Danang DIC, the effort and responsibility of all the contractors; the timely instruction and assistance of the World Bank’s task team members as well as the cooperation of all the beneficiaries are key to the success of the Subproject. 2. Many different types of beneficiaries involved make the project more complex and uncontrollable. 3. Various areas of expertise within IT projects make them difficult to implement since we cannot mobilize enough good experts within the limited budget of the Project. 4. Cost estimates for packages based on Vietnam ICT Norm is rather low and less attractive to qualified bidders. 5. Integration of all the investment systems to ensure their synchronization in operation is extremely difficult for any IT project. 6. The procurement process is often too long (one to two years) given rapid changes in technology and solutions. This leads to delay in contract implementation since several components originally stated in the bidding documents must be adjusted and re-submitted to the Bank and Local Authority for approval. Lessons Learned: Due to the complexity and particularity of IT projects, the Danang Subproject has gone though many difficulties, obstacles, as well as opportunities. Below are some of the key lessons to be drawn from the success of the Danang Subproject. 1. A project can achieve success at the end only if the project manager, beyond professional know-how and technological competence, is not motivated by self-interest but by the interests of the peoples’ success. He must be powerful enough to wield his influence and persuasion upon local leaders of different levels to gain approval for the project as well as to facilitate project implementation. 2. The close cooperation between Danang PIU and all their beneficiaries during project implementation. 3. Transparency, competition and fairness during the process of selecting contractors must be strictly-applied to all packages. This will help award contracts to the winning bidder with real competence. 4. The beneficiaries must be informed of the components which will benefit them and get them involved in the Project from the first steps of the procurement process. 5. New technologies and new solutions must be applied to the Project, taking into account the key factors such as budget, operational costs and systems integration. 87 Letter of June 20, 2014 from MIC PCU with comments on the Bank’s draft ICR: 88 89 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders N/A 90 Annex 9. List of Supporting Documents Project Appraisal Document (Report # 32919-VN, August 12, 2005) Development Credit Agreement Mission Aide Memoires Implementation Supervision Reports (ISRs) Restructuring Paper dated June 26, 2013 Restructuring Paper dated January 12, 2010 Financial Management Supervision documents Procurement Supervision documents Internal Bank Memoranda Ministry of Information and Communication (MIC/PCU): Final Report on Implementation of the ICT Development Project, May 2014 Danang Study: Final Report to Danang Project Implementation Unit of ICT Development Project - DNG 14: Assessing the Status of Danang e-Government Implementation, Leading Associates with KDi Asia, 2014. Impact Assessment Study: Computerized Services Delivery Projects from India and Chile: Bhatnagar, Tominaga, Madon and Bhatia, World Bank IT Staff Working Paper 42147, November 2007 Report on Performance Assessment of the Communications Campaign for the Vietnam ICT Development Project: An Vien Telecommunications and Communications Company and Ministry of Information and Communications PCU, 2013 Project Performance Assessment Report: Consultancy Support on Monitoring and Assessing the Quality and Effectiveness of Implementation of all PIUs, KDi Asia, 2014 91