Document of The World Bank Report No: ICR00004042 IMPLEMENTATION COMPLETION AND RESULTS REPORT (ICR) ON A SMALL GRANT1 IN THE AMOUNT OF US$2.67 MILLION TO THE REPUBLIC OF ARMENIA FOR STRENGTHENING THE LIVELIHOODS AND VOICE OF POOR AND VULNERABLE PERSONS IN ARMENIA PROJECT February 15, 2017 Social Protection and Labor Global Practice Europe and Central Asia Region 1 This ICR satisfies any other requirement for small grants completion reporting. ii CURRENCY EQUIVALENTS (Exchange Rate Effective: January 1, 2017) Current Unit = Armenian Dram (AMD) AMD 487 = US$1 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS DRC Disability Resource Center FM Financial Management ICR Implementation Completion and Results Report ISR Implementation Status and Results Report JSDF Japan Social Development Fund MLSA Ministry of Labor and Social Affairs MA Mission Armenia M&E Monitoring and Evaluation NGO Nongovernmental Organization OTJ On-the-Job OM Operational Manual PDO Project Development Objective PSC Project Steering Committee PWD Person with Disabilities SC Steering Committee SEA State Employment Agency Regional Vice President: Cyril E. Muller Country Director: Mercy MiyangTembon Senior Global Practice Director: Michal Rutkowski Practice Manager: Andrew D. Mason Project Team Leader: Marina Petrovic ICR Team Leader: Maddalena Honorati iii REPUBLIC OF ARMENIA Strengthening the Livelihood and Voice of Poor and Vulnerable People in Armenia TABLE OF CONTENTS DATA SHEET A. Basic Information ......................................................................................................................1 B. Key Dates ..................................................................................................................................1 C. Ratings Summary ......................................................................................................................1 D. Results Indicators ......................................................................................................................2 I. Project Development Objectives and Design ......................................................................4 II. Key Factors Affecting Implementation ...............................................................................7 III. Assessment of Outcomes ...................................................................................................11 IV. Risk to Sustainability .........................................................................................................17 V. Lessons Learned.................................................................................................................18 ANNEXES Annex 1. Team Composition ........................................................................................................21 Annex 2. Implementing Agency Background ..............................................................................23 Annex 3. Outputs by Component..................................................................................................25 Map: IBRD 33364 iv DATA SHEET A. Basic Information Region: Europe and Central Asia Country(ies): Armenia Global Practice: Social Protection and Joint with IFC No Labor Recipient: Mission Armenia, on Implementing Mission behalf of the Government Agencies: Armenia of the Republic of Armenia Sector(s)/Theme(s): 100% Social Protection Theme Code (as % of total Bank financing) Income Support for Old Age, Disability & 30 Survivorship Other social protection and risk management 30 Participation and civic engagement 20 Social Safety Nets/Social Assistance & Social Care 20 Services Executed by: Recipient Project ID: P125791 TF Name and TF013894 Number(s): Original Grant Amount: US$2,670,000 Donor(s): Japan Revised Amount: US$2,670,000 Disbursed Amount: US$2,670,000 Environmental C Safeguards Triggered: n.a. Category: B. Key Dates Process Date Process Date Concept Review: 17-Dec-2012 Effectiveness: 15-Mar-2013 Appraisal: Restructuring(s): 19-Oct-2015 Approval: 01-Mar-2013 Closing: 01-Mar-2016 Revised Closing Date: (if 01-Sep-2016 any) C. Ratings Summary Outcomes: Satisfactory Implementation: Moderately Satisfactory Risk to Sustainability: Moderate 1 (a) Project Development Objective Indicators2 D. Results Indicators Original Target Actual Value Formally Values Achieved at Indicator Baseline Value Revised Target (from approval Completion or Values documents) Target Years Indicator 1: Number of graduates in sustained wage employment (Number, Custom) Value — (Quantitative or 0 200 203 Qualitative) Date achieved May 17, 2013 March 1, 2016 — August 31, 2016 Comments (including % Target surpassed. achievement) Indicator 2: Number of businesses in sustained operation (Number, Custom) Value (Quantitative or 0 40 — 90 Qualitative) Date achieved May 17, 2013 March 1, 2016 — August 31, 2016 Comments (including % Target surpassed. achievement) Increase of number of individuals helped by Mission Armenia to get benefits Indicator 3: (Number, Custom) Value (Quantitative or 88 1,000 — 1,239 Qualitative) September 30, — Date achieved March 1, 2013 August 31, 2016 2013 Comments (including % Target surpassed. achievement) Retaining or increasing capacity to independently carry-out activities of daily Indicator 4: living (Percentage, Custom) Value (Quantitative or 60 70 — 85 Qualitative) September 30, — Date achieved March 1, 2016 August 31, 2016 2013 2 These are Project Development Objective (PDO) indicators that are tracked in the ISRs. For further explanation see Section 1.1. 2 Target surpassed. This indicator is clarified in the Operational Manual (OM) Comments as: ‘Beneficiaries retain or improve their well-being, as measured by (a) (including % Morbidity per month; (b) Increase of capacity to independently carry-out achievement) activities of daily living; and (c) Increase of number of individuals helped by Mission Armenia (MA) to get benefits’. Indicator 5: Community financing (Percentage, Custom) Value (Quantitative or 10 11 — 12 Qualitative) Date achieved May 17, 2013 March 1, 2016 — August 31, 2016 Comments Target surpassed. Community financing is defined as contributions from local (including % governments, individuals, and from other nongovernmental organizations achievement) (NGOs). Indicator 6: Visits to project-supported government facilities (Number, Custom) Value (Quantitative or 0 n.a. — 19,666 Qualitative) Date achieved May 17, 2013 March 1, 2016 — August 31, 2016 Comments This indicator was linked to rehabilitation of infrastructure, which was ongoing (including % till the project closing date. Number is an estimate. achievement) (b) Intermediate Outcome Indicator(s) Original Target Formally Actual Value Values Revised Achieved at Indicator Baseline Value (from approval Target Completion or documents) Values Target Years Indicator 1: Number of beneficiaries. (Number, Custom) Value (Quantitative 2,100 4,000 — 4,856 or Qualitative) Date achieved September 30, 2013 March 1, 2016 — August 31, 2016 Comments (including % Target surpassed. achievement) 3 I. Project Development Objectives and Design 1. Original Project Development Objectives (PDO) and Key Indicators 1. The approval for this Japan Social Development Fund (JSDF)-funded project did not include the drafting of a separate Project Paper. Therefore, this Implementation Completion and Results Report (ICR) evaluation uses the PDO statement from the Grant Agreement (GA), Operational Manual (OM), and Implementation Status and Results Reports (ISRs): To pilot self- reliant livelihoods and generation of employment opportunities combined with the provision of vital services for extremely poor and vulnerable groups, including persons with disabilities, in Armenia. 2. An initial draft set of indicators was included in the Grant Proposal. Later on, these were elaborated and approved as part of the OM for the project and began to be tracked in the ISRs. For this completion evaluation, the list of key indicators from the ISRs (shown below) will be used. These are considered consistent with the Legal Agreement and with the other project documents. (a) Number of graduates in sustained wage employment (b) Number of businesses in sustained operation (c) Increase of number of individuals helped by Mission Armenia to get benefits (d) Retaining or increasing capacity to independently carry-out activities of daily living (e) Community financing (f) Visits to project-supported government facilities. 3. The indicators from the ISR deviate slightly from the OM indicators. The OM indicators are thus also reviewed in the ICR, as additional evidence. Table 1 compares the outcome indicators from the ISRs with the indicators from the OM. Table 1. Results Framework - Comparison between Indicators in the ISRs and OM Outcome Indicators Outcome Indicator Target Target Achievement from ISR from OM Number of graduates 200 At least 200 of the 200/700 203 graduates in wage in sustained wage 700 graduates of employment employment (Number, training courses are Custom) still in wage In addition: employment three  about 800 unemployed months after they poor and vulnerable people were placed in jobs. start training. About 70 percent are person with disabilities (PWDs) and about 30 percent are other vulnerable people;  about 550 of these poor and vulnerable people graduate from training; and  out of the 800 people 4 Outcome Indicators Outcome Indicator Target Target Achievement from ISR from OM starting training, about 150 unemployed poor people and PWDs receive on-the- job (OTJ) training. Number of businesses 30 At least 40 of the 80 40/80 90 businesses in operation in sustained operation new businesses are (Number, Custom) still in operation In addition, out of the three months after approximately 700 graduates of vocational training, about start-up. 170 graduates receive additional training to start micro-businesses. Retaining or 70 Beneficiaries retain — 86.5 percent (based on increasing capacity to or improve their individual assessment of the independently carry- well-being, as indicators for each out activities of daily measured by beneficiary) living (Percentage,  morbidity per In addition, Custom) month,  increased capacity  15 multifunctional service to independently centers that deliver carry out activities services to poor and of daily living, vulnerable people, and including PWDs, were rehabilitated; and  increased number of individuals  social workers help 80 helped by Mission percent of beneficiaries Armenia (MA) to gain social benefits, get benefits. participate in public programs, acquire disability devices (such as wheelchairs and hearing aids), receive public documents, obtain social assistance, and exercise entitlements (such as free medical care) and social rights. Increase of number of 1,000 — — 1,239 individuals helped by Mission Armenia to get benefits (Number, Custom) Community financing 11 Community 10 12 (Percentage, Custom) financing of services to these beneficiaries In addition, rises by at least 10  4,000 beneficiaries receive percent over the 2.5 community-based social 5 Outcome Indicators Outcome Indicator Target Target Achievement from ISR from OM years following and health services, and all effectiveness of the 4,000 beneficiaries receive grant. social services. Of these, 3,000 receive health services, 500 bedridden beneficiaries receive services at home, and 2,050 receive hot meals every day. Visits to project- 10,000 10 percent increase 10% 19,666 supported government in visits by PWDs facilities (Number, and poor people to Custom) government offices: (a) rehabilitated by the project and (b) where the project delivers service. (Percentage) PWDs receive more No target No data on number of information that beneficiaries. helps them make decisions about their 12 public sites are remodeled. future. Poor and vulnerable No target About 500 poor PWDs and people, including other poor and vulnerable PWDs, increase their people receive training to participation in local advocate for themselves and government and help themselves (from final public events in the quarterly report). 15 communities. Intermediate Target Intermediate Target Achievement Indicators from ISR Indicators from OM Number of 4,000 — — 4,856 beneficiaries (Number, Custom) 2. Main Beneficiaries 4. The project aimed to support about 4,000 poor PWDs and other extremely poor people who live in 15 urban locations in six Armenian provinces. The beneficiaries included the following sub-groups:  800 receiving vocational and on-the-job training and micro-business opportunities  4,000 receiving vital social, health care, and nutrition services  700 with access to information and advocacy education 6 5. The beneficiaries were selected based on specific criteria decided and agreed by the Project Steering Committee (PSC), which included the Ministry of Labor and Social Affairs (MLSA), disabled people’s organizations, and local government representatives among others (see section 3). The criteria are described in the OM and include: (a) poverty status; (b) disability status; (c) gender equality; and (d) trainee’s willingness to contribute to the training fee as a proof of commitment and motivation to participate. Additional criteria were used to select beneficiaries for Components 2 and 3. 6. According to the OM, the definition of poverty and extreme poverty was aligned with Armenia’s National Statistical Service, which at the time of preparation defined as poor those with consumption per adult equivalent below the poverty line (AMD 33,517 per month) and as extreme poor those with consumption per adult equivalent below the food extreme poverty line (AMD 19,126 per month). MA’s project staff home visits were used as a way to verify beneficiaries’ eligibility.3 7. The OM, together with the Financial Management (FM) Manual, describes all procedures that were to be applied by the MA, a non-governmental organization (NGO), during project implementation. 8. In parallel, the MA—the recipient and implementing agency—benefits from seed financing and capacity-building opportunities. II. Key Factors Affecting Implementation Project Cost (a) Project Cost by Component (in US$, million equivalent) Appraisal Estimate Actual/Latest Estimate Percentage of Components (US$) (US$) Appraisal 1. Provision of Vocational Training and 887,430 883,162 99.52 Boosting Self-Employment 2. Provision of Vital Social Care and 600,560 592,081 98.59 Services 3. Creation of an Inclusive Environment for People with Disabilities and 591,060 584,651 98,92 Enhancement of their Self-advocacy 3 At the time of project preparation, based on the Armenia Integrated Living Conditions Survey of 2010, 36 percent of population lived in poverty. Totally 177,035 persons were officially registered as PWDs in Armenia: of these, 64 percent were working-age adults (16 to 63 years old). Estimates showed that only about 12 percent of working-age PWDs were employed, compared to 80 percent among all working-age adults. Most PWDs depend on disability benefits that vary from about US$30 to US$40 per month, which is lower than the average non-disability pension of US$66. 7 Appraisal Estimate Actual/Latest Estimate Percentage of Components (US$) (US$) Appraisal 4. Monitoring, Evaluation, and Project 590,350 581,763 98,55 Management Total Project Costs 2,669,400a 2,641,657b 98.96 Note: (a) the column ‘Appraisal Estimate’ shows the component costs as given in the Grant Proposal. The arithmetic total of Components 1, 2, 3, and 4 is US$2,643,800. However, the grant total is US$2,669,400. This may be due to a technical error in the Grant Proposal; and (b) this total number is from the World Bank system after final reconciliation and is slightly higher than the exact sum of the component costs as reported by the recipient at closing (US$2,641,746). (b) Financing Appraisal Estimate Actual/Latest Estimate Percentage of Source of Funds (US$) (US$) Appraisal IBRD Grant 2,669,400 2,641,657 98.96 9. The original Grant Proposal mentions an expected government contribution of about US$1.7 million ‘to support the proposed activities’. The Government’s commitment was reportedly met by the spending of US$1.6 million in the form of salaries for MA local staff, office utilities, and other nonrecurring costs.4 Original Components 10. The grant funding was provided to MA for the Strengthening the Livelihoods and Voice of Poor and Vulnerable Persons in Armenia Project. This three-year project sought “to pilot self- reliant livelihoods and generation of employment opportunities combined with the provision of vital services for the extremely poor and vulnerable groups, including persons with disabilities, in Armenia.� About 4,000 extremely poor persons, particularly, PWDs of different age and gender groups in six Armenian provinces (Yerevan, Shirak, Lori, Ararat, Kotayk, and Gegharkunik), were expected to benefit from project activities under its four components. 11. The project had four components as originally designed. 12. Vocational and Self-employment Training. Vocational training was delivered in the form of class-based and OTJ training to the extremely poor and vulnerable groups, including PWDs. The self-employment sub-component was aimed at boosting business creation by providing technical (business training) and financial support (small sub-grants) to help selected graduates of the vocational training jump-start micro-business and income-generating activities. Activities under Component 1 were delivered by different providers. 4 The figure is slightly lower because of three-year deflation (in case of calculation in Armenian Dram the Government’s co-investment is even higher than planned). 8  MA: Vocational training courses on specific trades5  Other NGOs, private firms and colleges, on-the-job training, or public institutions: training courses and micro-business opportunities 13. Vital Social and Health Services. The project supported the operation of community- based social and health care services for the extremely poor and vulnerable, including homebound and disabled people. MA delivers basic services to PWDs and other poor and vulnerable groups in their communities rather than at traditional large residential institutions. Under Component 2 of the project, it was intended to deliver social services, such as counseling, to about 4,000 beneficiaries, primary health services to about 3,000 of the beneficiaries, hot meals daily to about 2,050 of the beneficiaries, and health care home services to about 500 bedridden beneficiaries. 14. Access to Information and Advocacy Education. The project supported the establishment of disability resource centers (DRCs), improvement of environmental and communication accessibility for PWD, and empowerment of the beneficiaries with self-advocacy skills to independently enforce their rights and privileges. Component 3 was intended to help PWDs become more independent, address, and resolve some of their own difficulties. To achieve this, the component supported the following activities:  Better access to public facilities and to public information (a) Establishment of 12 DRCs by renovating and remodeling parts of multifunctional service centers (day care centers) for the dissemination of information to PWD and others (b) Remodeling of 12 public buildings so that PWD could gain access; obtain information; and apply for documents, services, and benefits  Advocacy education (a) Skills training in advocacy and self-help (b) Creation of self-help groups 15. Monitoring, Evaluation, and Project Management. The MA was responsible for the monitoring and evaluation (M&E) of the project and for reporting on the indicators to the PSC and to the World Bank. Elements of the M&E system included: (a) regular field visits by the MA to ensure that services, goods, and works were actually delivered and the quality of services, such as vocational training, and monitor contractors’ respect for the terms of their contracts with MA; (b) the creation and update of a database to track progress of the project, including information on inputs, outputs, and outcome indicators; (c) the preparation and administration of questionnaires on beneficiary achievement, satisfaction, and proposals for improvement of training and social services; (d) MA social workers’ reports on a sample of beneficiaries of 5 Hairdressing, manicure, cosmetics and makeup, table decoration, embroidery, knitting/carpet weaving and computer courses. 9 health and social care services; (e) quarterly progress reports; and (f) a baseline and final evaluation report out-sourced to external consultants. 3. Summary of Implementation and Significant Changes Rating: Moderately Satisfactory 16. The MA, a national NGO, approached the World Bank for assistance in mobilizing funds for a project for vulnerable people, PWDs, and other extremely poor people. To support this request, the Ministry of Finance of Armenia asked the World Bank to apply to the JSDF to fund this project. The Ministry of Finance of Armenia endorsed the MA as the implementing agency with the understanding that a PSC chaired by the MLSA would be established at project inception to oversee the project implementation. The MLSA, the State Employment Agency (SEA), and other stakeholders were partners to the process. 17. The PSC, chaired by the Deputy Minister of Labor and Social Affairs, was established with representatives of public and nongovernmental sectors to ensure the Government’s and broader public participation in the design and implementation process of the pilot. The PSC included the Ministry of Territorial Administration, the Ministry of Finance, the MA Project Director, representatives of Development Partners Organizations, NGOs, and representatives of communities (local governments). 18. The project was formally restructured once, in October 2015, to extend the closing date. The project’s original closing date was March 1, 2016. At the request of the implementing agency and after a review by the World Bank team, it was decided that a six-month extension would be granted to allow for the completion of the remaining activities, including monitoring of small businesses and the final project evaluation. To evaluate the impact of project activities and its PDO, it was necessary to track beneficiaries’ employment status three months after the completion of the program, which spanned beyond the project life. 19. Throughout implementation, the project’s OM was revised several times to reflect emerging needs. Neither the number of project sites nor the targeted number of beneficiaries was affected by the introduced changes. The Results Framework, as reflected in the OM, was adapted during the life of the project to provide a better measure of intended outputs and outcomes. The changes modified the wording but did not change the intent of the indicators being measured. The World Bank team held internal consultations (including the JSDF Unit and Legal Department) and decided that no restructuring was needed. Therefore, the OM was amended by the recipient with a ‘No Objection’ from the World Bank. 4. Safeguard and Fiduciary Compliance 20. The project had no reported safeguards issues during the course of implementation. Procurement and FM activities are found to be Satisfactory, with some exceptions as highlighted in the following paragraphs. 10 21. Procurement administration. The procurement of goods/works and selection of consultant services (including the training) were implemented in line with the agreed rules and provisions. The procurement filing system was assessed by the World Bank as Satisfactory. However, the involvement of procurement staff in contract administration was quite limited. The procurement specialist was not involved in reviewing and overseeing procurement-related aspects of the sub-grant projects. 22. Procurement capacity. Adequate fiduciary support was provided by qualified procurement and financial specialists. One full-time procurement specialist had experience in World Bank-funded projects and adequate capacity for implementing small-value procurement/selection. 23. Procurement plan/post review. The latest updated procurement plan before project closing contained 172 signed contracts (the majority of contracts were signed using shopping procedures included with numerous lots in packages) for goods and 85 contracts for consulting services and training. During the recently conducted procurement post review mission, six contracts were reviewed and five project sites (within Yerevan City) were visited. The post review report has been posted on the project website. The performance and compliance risk was assessed as Low while the ‘contract management’ was assessed as Moderate. The last procurement review does not seem to find major issues, but it does record some concerns related to quality of works and engineering design that were to be remedied by extending some construction contracts. 24. Complaints. During the review period, no procurement-related complaints were received/recorded. 25. Financial Management. Overall, FM was Satisfactory. However, certain recommendations made during the life of the project on improving the accounting software (specifically to address issues with applying accrual basis of accounting through presentation currency and automatic revaluation of monetary items) were implemented only partially. In addition, reviewing the contract forms and complying the purchased goods and property with nature and terms of the contract has not been implemented. III. Assessment of Outcomes 1. Achievement of Project Development Objectives 26. In the absence of a Project Paper, there are three documents that contain versions of the key performance indicators—the JSDF Grant Proposal, the OM cleared by the World Bank, and the ISRs. For the purposes of this ICR, indicators from the ISRs will be used to evaluate project outcomes6 with the indicators from the OM providing supplemental information. The indicators from the Grant Proposal are considered concept-stage indicators and are therefore not being evaluated. Table 1 in Section 1 provides a comparison between the ISR indicators and the OM indicators. Some of the OM indicators correspond directly to ISR indicators and others are 6 The ICR evaluation team consulted the World Bank’s operational policy unit in taking this decision. 11 additional. The indicators do not provide disaggregation by poverty and vulnerability status; however, the disaggregation is possible through the project’s M&E system. Assessment of Relevance: Substantial 27. The relevance of the PDO was High at the time of the project design and remains High at the time of completion. The piloting of services, including social and employment, for disabled and poor people is very important for the development of the Armenian social protection system. In addition to that, the pilot shows the success of an NGO-delivered social service model, which is widely used around the world, and in Europe in particular. In addition, given the small project size and short implementation period, the PDO was sufficiently ambitious. 28. The PDO was in line with the Country Partnership Strategy agreed between the Government of Armenia and the World Bank at the time, with one of the overarching objectives being to address vulnerabilities of the population. Vulnerable groups, including PWDs, are less informed about their rights and privileges and lack skills for self-advocacy. The project aimed to assist the social and labor market integration of the extremely needy and disabled individuals. 29. The project is also in line with the current World Bank Country Partnership Strategy for Armenia FY14–FY17. 7 The Country Partnership Strategy supports the Government's overarching objective of job creation. Within this framework, there are two clusters of outcomes and the project is linked to both: (a) Supporting competitiveness and job creation; and (b) Improving efficiency and targeting of social services. 30. The relevance of design is also High. The project components were carefully selected to pilot a range of services and provide the much-needed information on demand for and on the implementation of delegated services within the Armenian context. The components were designed to address challenges related to limited access to and information on available services for PWDs, lack of sound institutional mechanisms for employment of poor people and PWDs, low levels of vocational and soft skills training, and poor awareness among many PWDs of their rights and fragmented nature of efforts. The project design incorporated recommendations provided in the World Report on Disability (World Health Organization and the World Bank 2011). 31. With regard to addressing employability constraints, the project design piloted innovative approaches to ‘activate’ poor people and PWDs, traditionally perceived as passive recipients of social assistance benefits by: (a) enhancing their technical and entrepreneurial skills; (b) promoting inclusive environments through the establishment of DRCs; and (c) helping them advocate for their rights. 32. Finally, the relevance of implementation is Substantial. The NGO-based implementation model adopted by the Government of Armenia for this project represented an innovative and successful part of the PDO achievement. There were two discernable periods in the project 7 World Bank. Country Partnership Strategy for the Republic of Armenia (October 9, 2013). Report No. 81647-AM. 12 life—an initial period, characterized by somewhat slower implementation, and a second period, when implementation picked up and, eventually, planned activities were largely completed and of good quality. 33. The project became effective in March 2013 and started with some moderate delays in implementation. The reasons included: (a) a period during which the MA had to hire qualified staff; (b) the need to train the MA staff on the World Bank’s rules and procedures; and (c) a period of adjustments related to the management structure and delegation of authority within the MA. Once these issues were resolved, the implementation of project activities picked up in 2014. The social services activities (Component 2) gained speed more quickly, and starting in 2013, they were provided adequately and on a continuous basis within the MA's multifunctional service centers and in beneficiaries' homes. These services included the provision of hot meals, health services, social care services, counseling, and psychosocial support. 34. The implementation of vocational training and job-search activities (Component 1) gained speed in 2014, and eventually the project made significant progress in the second half of the implementation period. At the request and with the support of the World Bank, the MA hired proactive training experts for vocational and on-the-job training activities, as well as for the implementation of the small grants/self-employment schemes. The World Bank brought on board additional staff to support the MA in steering the training and employment activities, while maintaining a significant level of implementation support. Most of the trainings offered were computer literacy courses, as well as courses aimed at benefitting women as a primary vulnerable group. In general, the World Bank team’s support during the implementation phase was key for the successful turnaround of the project and for the successful end result. 35. Actions in specific areas were pushed during the last months of implementation to meet a full set of defined targets—to complete remodeling of designated facilities, increase their use (Component 3), and finalize the evaluation of project activities (Component 4). Annex 3 in this ICR provides a detailed account of the project’s outputs. 36. Overall, the implementation of the specific formal program activities followed the initial plans, stayed relevant to the evolving policy context in the country, and was conducive to producing the desired outputs. There was no major policy shift that would have required significant changes to the project implementation or design. The implementing agency was diligent in contracting the various trainings, enrolling the required number of participants, and delivering the planned services. 37. On the learning side, it is worth noting that some aspects related to sustainability and policy development could have been pursued more rigorously. For example, territorial agencies of the MLSA remained relatively distant from project activities, and more work would be needed to define if and how the pilot’s experience could be integrated into a more permanent (possibly budget-financed) program. The PSC managed to provide the necessary authorizing environment, but it could have gone beyond that to try and create the required policy change. 38. Similarly, the links to the SEA services and programs could have been stronger, and more cooperation with the Armenian Social Investment Fund in the implementation of Component 3 would have been useful. However, it is recognized that a small pilot could not be expected to 13 lead to full sustainability and policy advancements. Therefore, the overall combined relevance of PDO, design, and implementation is assessed as Substantial. Assessment of Efficacy: Substantial 39. The efficacy of the project was found to be Substantial. 40. Overall, all six key performance indicators were achieved, with five indicators, in fact, surpassing their original targets. The achievement of outcomes (by PDO indicator) at project closure was the following: (a) Number of graduates in sustained wage employment—target 200, achieved 203 (including 92 self-employed, 41 informal employees, and 70 formally registered employees) out of 845 vocational training graduates. It is to be noted that about 84 percent of graduates are PWDs—target achieved. Table 2 shows the high training program take-up (95 percent of enrolled beneficiaries actually complete and graduate from the training courses); 24 percent of the training graduates found a job and are still employed three months after vocational training completion (those who received training OTJ rather than class-based are more likely to be employed), and one in three of those employed are actually formally registered employees. (b) Number of businesses in sustained operation—target 30, achieved 90 small businesses still in operation as of August 2016—target surpassed. It is to be noted that among the vocational training graduates who were selected to participate in the small business grant project sub-component, almost half of them were involved in jump-starting a micro-business. (c) Retaining or increasing capacity to independently carry-out activities of daily living (Percentage, Custom)—target 70 percent, achieved 86.5 percent—target surpassed; (in accordance with the Armenian legislation, this indicator refers to self- service capability maintenance level). During the project lifetime, 633 beneficiaries received home care services, and 161 (about 25 percent of 633) had activities of daily living increase (increasing capacity to independently carryout activities of daily living). This is added to the baseline of 60 percent. (d) Increase of number of individuals helped by MA to get benefits—target 1,000, achieved 1,210 individuals helped (compared to baseline 88 and target 1,000)— target surpassed. (e) Community financing (Percentage, Custom)—target 11 percent, achieved 12 percent. (Community financing is defined as contributions from local governments, individuals, and from other NGOs) to social services to PWD—target surpassed. (f) Visits to project-supported government facilities—target 10,000, achieved 19,666 beneficiary visits to project supported facilities—target surpassed. 41. In addition, 18 social service sites were remodeled to improve the accessibility of the services to PWDs. Information about the available services was also disseminated to them, 14 although the exact number of people who received this information was not tracked. There is anecdotal evidence of increased participation of PWD in local government and public events, although the number was not tracked. There were 45 trainings on advocacy and self-help for 430 PWDs, however, which likely contributed to increased participation. 42. Finally, there is one intermediate outcome indicator which is a Core Corporate Results Indicator: The number of project beneficiaries. The number of beneficiaries of the vital services, the project has exceeded its target of serving 4,000 persons—4,856 beneficiaries received the vital services (nutrition, health, and social) during the project (as of end of August 2016). 43. While a final project evaluation was conducted, it did not provide in-depth information on the outcome indicators but rather relied on monitoring data from the MA and barely mentioned the final achievements in relation to the targets. Instead, this evaluation functioned more as a beneficiary assessment, providing insights on the beneficiaries’ attitudes and satisfaction with the program. Overall, in support of the achievement of the outcomes, this evaluation reveals high levels of beneficiary satisfaction with the various program interventions. For example, related to Component 1, the evaluation shows that the overall satisfaction of the respondents with the vocational and OTJ trainings was high (mean score = 5.5 out of 6). Out of 51 surveyed participants of the training courses who were employed at the time of the survey, the majority (75.5 percent) had been in wage employment three months and more. Furthermore, 35.3 percent reported that the training program helped them obtain a job and 61.1 percent of them were still in wage employment three months and more after being placed in the job. 44. For Component 2, where many beneficiaries are bedridden, the evaluation revealed that the majority of interviewed beneficiaries (97.6 percent) have reported using the MA health services at home. Nearly 95 percent of them expressed their satisfaction with the services. In total, 91.7 percent of interviewed beneficiaries wished to continue using the health services provided by the MA. Table 2. Project Outcomes (Component 1)8 Employed Three Formally Number of Enrolled Months after Registered Graduates Graduation Employee Vocational training graduates 894 845 203 70 OTJ training 151 52 Class-based vocational training in MA DRCs 246 Class-based vocational training in 59 education centers 448 Self-employment training 210 205 92 Source: MA monitoring reports. Note: Ninety-two beneficiaries were involved in launching 90 businesses. 8 See Annex 3 for detailed disaggregation of outputs by location. 15 Assessment of Efficiency: Substantial 45. The project’s efficiency is assessed as Substantial. Despite the initial delays, the project achieved and surpassed all outputs and outcomes cost-effectively, especially given the vulnerability nature of the targeted beneficiaries (disabled, the poor, mostly women who have higher constraints entering the labor market in Armenia). The overall costs were aligned with the planned costs at appraisal. Looking solely at the administrative costs, those stood at around 17 percent (excluding M&E costs), or 22 percent, including ongoing M&E reporting and project assessment, which are slightly higher compared to the average social protection projects. However, considering that this is a pilot project and that administrative costs are higher during the early stage of implementation and lower for mature programs, the share of the administrative costs of total project costs are aligned with international experiences. 46. As to efficiency of the program itself with regard to cost per beneficiary—or economic rate of return—there was not enough information to quantify benefits (with regard to salaries of beneficiaries employed three months after program graduation or other monetary gains). However, the unit cost per beneficiary for six months vocational training (about US$450) compares favorably with similar programs.9 The ICR could not apply a cost-benefit analysis to assess the efficiency of the vocational training sub-component by comparing monetary monthly salaries of program participants with total unit costs (delivery and administrative costs). The self- employment sub-component has the highest unit costs due to the high value grant awarded (about US$4,300). This is higher than similar programs in Armenia—however, those were not targeted specifically to PWDs, such as the small business grant program under the state budget and the grant under the ongoing World Bank Social Protection Administration Project II which plans to start awarding US$3,000 to selected business plans. When accounting for the fact that supporting PWDs is generally very expensive, the cost achieved by this project for self- employment compares well to the Armenian national experience. No business profit and sales data have been collected to allow a cost-benefit analysis of this sub-component, nor is it fair to say it is expensive compared to other programs given its unique nature of targeting PWDs. Table 3. Project Delivery Unit Costs, by Component Total Delivery Cost Total Cost per Participant (US$) Beneficiaries (US$) Component 1 Vocational training 407,941 894 456 Business training 79,261 205 387 Business grants 395,959 92 4,304 Component 2 Social health services for the vulnerable 96,778 4,856 20 Nutrition services for the vulnerable 495,303 2,948 168 9 US$478 per beneficiary in Kenya (Honorati, M. 2015. “The Impact of Private Sector Training and Internship on Urban Youth in Kenya�, Policy Research Working paper 7404, World Bank, Washington, D .C.), and similarly in Colombia (Attanasio O. P., A. D Kugler., and C. Meghir. 2011. "Subsidizing Vocational Training for Disadvantaged Youth in Developing Countries: Evidence from a Randomized Trial," American Economic Journal: Applied Economics 3:188-220 2011). 16 Component 3 Advocacy and self-help training 12,997 2,150 6 Environmental and accessibility for 571,654 — 28,583 PWD** Source: MA monitoring reports. Note: **Refer to total operating costs for the remodeling of public infrastructure and rehabilitation of DRCs; as of August 2016, 20 sites have been remodeled/rehabilitated. The number of training beneficiaries refer to ‘enrolled’ beneficiaries who attended at least one day of training, not those who successfully graduated. 47. Overall Outcome Rating: Satisfactory. Based on the substantial relevance, efficacy, and efficiency, the overall outcome rating for the project is Satisfactory. 2. Other/Unintended Outcomes and Impacts 48. The project had a number of additional impacts that were not directly reflected in the PDO:  The project demonstrated the potential positive features of a service delivery model through an NGO. The final beneficiary surveys showed very good results and acceptance on behalf of the beneficiaries. The project provided important design and implementation lessons on the replicability of the pilots by government agencies and nongovernmental entities, especially the delivery of skills training and small business grants and the provision of social, health, and care services through case workers.  The project also promoted an inclusive environment where the disabled are more integrated with the non-disabled. The increased inclusiveness on the part of non- disabled in accepting disabled as an active part of society is an important element of inclusion. This perspective is confirmed by the final evaluation report.10  MA day care centers (‘multifunctional service centers’) (Component 2) tested a model of individualized case management for vital social care and services. The MA deploys a multifunctional team at each center to serve a cluster of beneficiaries. The team includes a social worker, a care provider, a doctor, a nurse, a team leader, and soup kitchen staff. This model is being considered for adoption and expansion by the Government (more details on the model can be found in Section V, Lessons Learned). IV. Risk to Sustainability Rating: Moderate 49. The risk to sustainability is assessed as Moderate. The project managed to demonstrate the success of a model through piloting social services and labor market programs through an NGO, MA. It has to be recognized that the pilot has created a momentum that is likely to result 10 American Center of Integration and Democratization (ACID). 2016. Strengthening the Livelihood and Voice of Poor and Vulnerable People in Armenia: Quantitative and Qualitative Evaluation Report. 17 in some form of continuation of the program. The part of the PDO referring to provision of vital services is already reasonably sustainable. The MLSA finances transfers to the MA on a regular basis for these services and the funding is included in their annual budgets. Overall, the pilot achieved its intended outcomes and contributed to the sensitization of the Government on the challenges of delivering skills training and small business grants for vulnerable groups, including PWDs. While there are no specific plans for immediately integrating these employment services into a permanent government-funded program, the pilot provided important design and implementation lessons on its replicability by both government and nongovernment agencies. Risk to sustainability is assessed as moderate as there is still no tangible evidence of the project’s direct contribution to policy making, although it has to be noted that at the design stage the expectation was to pilot and learn from the MA approaches to employability rather than institute the pilot into a sustainable government-funded project. V. Lessons Learned Policy-level Lessons (a) Piloting policy approaches through NGOs provides a viable opportunity for piloting and learning, while ensuring government involvement. With the right amount of coordination and financing, social services delegated to NGOs demonstrate viability, flexibility, and the potential to expand faster than government-implemented services. Initial plans involved the expectation for close cooperation between the implementing agency and Government institutions such as the Ministry of Finance and the MLSA. In July 2013, a PSC was formed, chaired by the MLSA, and involved representatives of the Ministry of Territorial Administration, the MA, and local government representatives. This connection could be further strengthened on the ground in the future implementation of a delegated-services model. It is very important to ensure that there is adequate government leadership and ownership to ensure sustainability. Nonetheless, the pilot contributed to the policy dialogue around active labor market programs and policies and sensitized the Government on the viability of such innovative employability approaches (b) Changing the attitude of society is important for the success of programs targeted to PWD. Integration of vulnerable persons as fully active members of society leads to the necessity of addressing several issues starting from changing mentality with regard to perception of extremely vulnerable people as members of society who have equal rights and opportunities, as well as improving access of PWDs from the perspective of their physical capabilities. Public awareness campaigns and education programs at a young age, through schools, could be the way forward. (c) Establish transparent procedures to certify disabled and simplify the administrative burden for PWD. The evaluation studies and results of the beneficiary surveys concluded that the main challenges for the poor and disabled 18 were heavy bureaucracy, corruption, and discrimination toward PWDs. 11 These challenges prompt the policy-makers to strengthen the legislation as needed and to better enforce the implementation of existing anti-discriminatory legislation. Efforts should be made to ease the administrative burden on PWDs and curb administrative- level corruption by introducing transparent procedures, providing publicly all needed information, and so on. (d) Labor market programs for the disabled are costly. Policy discussions and program design of future interventions need to be clear on the high cost of the labor market interventions for vulnerable groups, especially for PWDs. Discussions within society need to be held continuously, to prepare the ground and explain why it is fair for society to bear these costs. Implementation-level Lessons (a) Engage employers in defining the competencies taught by training institutions. An evaluation study showed that participants of vocational trainings faced difficulties with finding a job. Many participants had no post-training opportunities to practice their new skills. Thus, to increase the employability of trained participants, it would be necessary to involve employers in the training phase, study employers’ demand, and possibly commit employers (as trainers or OTJ training providers) to certain minimum placement requirements. It is important to provide incentives for employers to work with training providers to more efficiently address the employment challenges of at-risk youth. Results-based or placement indicators could be included in the contracts to ensure that a certain percentage of the trainees get employed after the completion of training. (b) Post-training support and intermediation services are needed for increased impacts. A total of 39.2 percent of the training participants reported that they did not actively seek a job. This supports the lesson that the design of these de facto active labor market programs needs to incorporate work with the demand side (employers), as well as continuous post-training support and mediation to help the participants find sustainable jobs. (c) Ensure vital social care and services in beneficiaries’ homes and communities. The implementation of social care services under Component 2 presented a successful model for the implementation of professional, individualized, and case management-based social work. The advantage of the model is the community- based nature of the care rather than the use of specialized institutions, which tend to isolate the beneficiaries. The serviced people remain in their communities, in a familiar environment, where they can access a wide range of services. 11 Heavy bureaucracy. Public institutions, like notary, municipality, cadaster, passport department, health posts, and others are hard to access and do not share information readily. Corruption. Corruption has been blamed for hard access to certain services and to disability certification. Discrimination toward PWDs. Disabled people face discrimination when applying for jobs or when attempting to use public services. 19 (d) Advocacy is not enough, longer-term assistance is needed to support PWDs. Less than half of surveyed beneficiaries of Component 3 agreed with all statements regarding the role, capacity, and services of the centers. The core reason for the majority of beneficiaries who participated in advocacy trainings was to understand the rights of PWDs. This was identified as the major learning outcome of the training. However, most participants did not express an interest in becoming more proactive or advocating for themselves or others. More work is needed to understand why this happened and identify the needed support. (e) Ensure adequate implementation capacity early in the project and budget resources accordingly. With the NGO as grant recipient, advanced training and capacity building is needed to ensure quick start-up of implementation according to World Bank policies and procedures. Resources should be envisaged within the project for such capacity building—including the need for initial training in World Bank procedures—and implementation support. Indeed, projects combining basic services and employment services for vulnerable groups require very strong implementation support, which needs to be factored in the preparation and in the cost of the project. This project could leverage implementation support resources from other World Bank projects.12 (f) Pursue more involvement of the territorial agencies of the MLSA in future project activities. More work would be needed to define whether and how the pilot’s experience could be integrated into a more permanent (possibly budget- financed) program. The PSC also can play a further pivotal role in creating a needed policy change in the future. (g) Strengthen the links of similar projects to the SEA services and programs. T he job placement effort did not integrate into national programs—especially for the job placement in wage employment. While the MA staff members helped trainees with mediation services, prepared them for interviews, approached employers to inform them about the matching with graduates from the MA program, organized public- private discussions, and carried out advocacy trainings, the collaboration on the ground with the SEA on these efforts is not documented in M&E reports. The MA’s monitoring reports show numerous field visits during which the information on employment status was collected at the individual level. The MA and the SEA had discussions, but eventually their separate programs were implemented in parallel, with the SEA focusing on efforts to employ the disabled under the Government program. For future projects, a higher-level political agreement and coordination could help ensure the needed cooperation. 12 As a response, the project allocated resources for recruiting two fiduciary specialists (FM and procurement) with adequate knowledge on and experience of World Bank procedures to fill the gap. The two consultants were also supposed to provide OTJ training for the MA staff. In addition, country-based based procurement, FM, and social protection consultants of the World Bank provided daily implementation support. 20 (h) Ensure solid analysis of gender-barriers for skills development. Despite gender equality target in the selection of beneficiaries, 70 percent of beneficiaries were female. While employment outcomes (hence the demand for Active Labor Market Programs) are generally worse for women and while the MA traditionally serves mostly women, this gender unbalanced result suggests directing of outreach efforts and information awareness campaign for both men and women and tailoring of services provided to men-dominated trades, for example, in the case of vocational training (the majority of training courses provided were clearly women dominated). (i) Define delegation mechanisms in project management. Mechanisms for delegation of management responsibilities within the implementing agency should be outlined within operational documents (OMs), to ensure efficiency, continuity, and effective implementation. Annex 1. Team Composition Name Role Title Specialization Unit Location Marina Petrovic Task Team Senior Social Social GSP03 Washington, D.C. Leader Protection Protection Specialist Gabriela Counsel Counsel Legal LEGLE Washington, D.C. Grinsteins Mariana Counsel Senior Legal LEGAM Washington, D.C. Margarita Counsel Montiel Maddalena ICR Task Team Senior Evaluation GSP03 Washington, D.C. Honorati Leader Economist Josefina Posadas Team Member Senior Social GSP03 Washington, D.C. Economist Protection Armine Procurement Procurement Procurement GGO03 Yerevan, Armenia Aydinyan Specialist Specialist Arman Vatyan Financial Senior Financial GGO21 Yerevan, Armenia Management Financial Management Management Specialist Aleksan Team Member Senior Operations ECCAR Yerevan, Armenia Hovhannisyan Operations Officer Lusine Consultant Consultant Financial GGO21 Yerevan, Armenia Grigoryan Management Anna Goodman Team Member Program Administration GSP03 Washington, D.C. Assistant Satik S. Nairian Team member Program Administration ECCAR Yerevan, Armenia Assistant 21 Recipient Organization(s) Organization Role Contact Title Phone Email Name Mission Recipient and Hripsime MA +374 1 coordinator@ngo.missionn.am Armenia NGO main Kirakosyan President 444792 implementing agency 22 Annex 2. Implementing Agency Background 1. The implementing agency, MA, is a grass-roots organization that has over 20 years of experience, proficiency, and tangible track record in activities related to poverty reduction, vocational training, disability, and social partnership initiatives with the Armenian authorities. The overall motto and vision of the MA is to follow the ‘Break and Empower Concept’ while providing services to people in need. Since 1988, the agency has initiated and established innovative, beneficiary strength and empowerment based poverty alleviation programs in Armenia. 2. The MA is the founder and represents the Secretariat of the Black Sea NGOs Regional Network on Social Improvement and Sustainable Development, a founding member of the Armenia Civil Society Partnership Network of NGOs on Poverty Reduction, and a founder- member of the Caucasian Refugee/IDP NGO network. Through all its activities, the MA promotes protection of fundamental human rights of the vulnerable and advocates for social policy change. The MA pursues an overall strategy in compliance with the United Nations’ Millennium Development Goals, Madrid International Plan of Action on Aging, European Social Charter, and Sustainable Development Program of the Republic of Armenia. Active in this field since 1988, the MA has gained national and regional recognition for its innovative poverty- reduction programs, for being the advocate for the vulnerable, disabled, and the poor through community-based services provision, a variety of vocational training courses, and self- help/income-generating initiatives for different high-risk groups. 3. The MA has experience of working with PWDs. Since its establishment, it has been involved in different activities aimed at meeting emergency needs of PWDs and knowledge gap of the staff. Those activities include: (a) supporting people, affected by earthquake in 1988 and who became disabled, with ongoing care and acquisition of prosthesis, wheelchairs, and other devices; (b) creating job vacancies for PWD in the most devastated areas in Gyumri and Spitak for about 50 disabled women who had amputated limbs; (c) participating in training on occupational therapy and practicing among the caseload; and (d) conducting a public awareness campaign through the introduction of gesture-translation of television news for the audio challenged people; preparation of eight series of movies on PWDs titled ‘Full Life’; making posters, booklets, and books; and organizing round tables for different layers of the population (aimed at breaking the existing stereotype in the society toward PWDs). 4. The MA has sound financial systems in place. Financial administration is carried out in accordance with the Law on ‘Accounting Records’, the Republic of Armenia Act on the Standards of Accounting Records, and other legal acts. Regular financial reports are submitted to the management of the organization, as well as donor organizations and the taxation authorities of Armenia. Regular annual audits of the expenditure are conducted by certified auditing firms. The logistics/procurement procedures are based on the requirements of the donor and the legislation of Armenia. Overall, to date the NGO has received and administered over US$20 million in grants funding and contributions/donations, including a grant project of US$9.8 million from United States Agency for International Development and a US$2.0 million grant from the Government of Armenia. Other donors included Japan International Cooperation Agency; United Nations High Commissioner for Refugees; World Food Program; and Swedish, Swiss, British, Danish, and other overseas foundations. 23 5. The MA has implemented a program called ‘Rehabilitation of the Elderly Care Home in Syunik Region’ with a total amount of US$86,131 (March 4 through August 30, 2009). 6. The MA is the innovator, creator, executor, and promoter of the Community-based Social and Health Service Provision Model in Armenia and the Caucasus. The model offers standard procedures in conformity with international standards and technical principles of social work, concurrently taking into consideration the needs of the local population, as well as regional and cultural peculiarities. The MA follows international standards with regard to social, psychological, and legal assistance components. Moreover, it includes a variety of care services, vocational trainings, activation of potential abilities of the poor and disabled, and enhancement of self-estimation and self-assurance of the PWDs. The model has acquired national significance for promoting the betterment of social support mechanisms. The Armenian Government has incorporated the MA’s model in its strategic planning (State National Program, Decree #485) as a highly effective mechanism and model for social service delivery. 7. For its well-developed technical, management, and monitoring systems the Government of Armenia has licensed the MA’s infrastructures and services (RoA Certificate KHT 006, MLSA License). The MA has been granted an ISO standards certificate by the ‘Global Certification’ Company Limited, Certificate No.7562718343, which is the first precedent among the local NGOs. The MA has been awarded a medal by the MLSA for its contribution to the development and enforcement of the social support sphere in Armenia (Decree #2628-A/4). 24 Annex 3. Outputs by Component Component 1: Provision of Vocational Training and Boosting Self-employment 1. According to the project OM under Component 1 of the Project, 800 PWDs/unemployed persons were expected to increase their professional qualification and skills by completing vocational training programs. Vocational training was delivered in the form of class-based vocational training courses either in MA DRCs at educational centers, and through OTJ training to 845 beneficiaries in total, 84 percent of whom were PWDs. The selection of OTJ trainees was done taking into consideration the beneficiaries’ physical and mental abilities/constraints and the particular job’s conditions. The overall duration on the vocational training, in the different modalities, was three months. 2. Out of the 845 who received and successfully completed the three months of vocational training, 210 were selected for self-employment training and 205 successfully completed the training, prepared business plans, and submitted grant proposals. Cash grants roughly worth US$4,400 were awarded to the 92 most successful trainees and best proposals to initiate 90 small business activities. 3. Under Component 1,845 beneficiaries graduated from various vocational trainings till May 31, 2016. See tables 3.1 to 3.3. Figure 3.1. Total number of beneficiaries who graduated (successfully completed) the vocational training program Sl. No. Type of Course Male Female Total 1 Accounting training 5 7 12 2 Computer literacy 83 97 180 3 Computer operator 2 2 4 4 Confectionery 2 28 30 5 Cooking 31 74 105 6 Cosmetics/make-up 0 35 35 7 Embroidery/carpet weaving 3 13 16 8 Floristics 0 16 16 9 Furniture production 15 0 15 10 Greenhouse keeping 8 2 10 11 Hairdressing 33 103 136 12 Knitting/embroidery 0 6 6 13 Manicure 0 6 6 14 Massage 25 14 39 15 Metal-plastic doors and windows production 3 0 3 16 Metal-plastic doors and windows, entrance doors, 6 0 6 and furniture production 17 Printing 4 1 5 18 Shoe making 6 0 6 19 Silversmithing 7 0 7 20 Souvenir production 8 0 8 21 Table decoration 18 162 180 25 Sl. No. Type of Course Male Female Total 22 Tailoring service 0 10 10 23 Wedding accessories design 0 10 10 Total 259 586 845 Figure 3.2. Total number of trainees who participated in vocational trainings at the MA DRCs Only Number of Beneficiaries Target location Training type Completed Yerevan DRC Hairdressing 6 Yerevan Yerevan DRC Table design 6 Yerevan DRC Make-up/cosmetology 6 Gyumri DRC Table design 14 Gyumri DRC Hairdressing 12 Shirak marz Gyumri DRC Make-up/cosmetology 6 Gyumri DRC Knitting /embroidery 6 Artik DRC Hairdressing 8 Vanadzor DRC Computer literacy 21 Alaverdi DRC Computer literacy 8 Lori marz Alaverdi DRC Hairdressing 7 Alaverdi DRC Table design 13 Spitak DRC Computer literacy 8 Gavar DRC Computer literacy 7 Gavar DRC Carpet weaving/embroidery 5 Gavar DRC Table design 7 Chambarak DRC Table design 11 Gegharkunik Chambarak DRC Embroidery/carpet weaving 6 marz Chambarak DRC Computer literacy 14 Sevan DRC Table design 13 Sevan DRC Hairdressing 13 Sevan DRC Make-up/cosmetology 7 Sevan DRC Computer literacy 12 Hrazdan DRC Embroidery/knitting 5 Hrazdan DRC Table design 8 Kotayk marz Hrazdan DRC Computer literacy 11 Hrazdan DRC Hairdressing 6 Total 246 Figure 3.3. Total number of OTJ trainings and trainees by locations Training Number of Beneficiaries Target Location Training Type Completed 26 Training Number of Beneficiaries Target Location Training Type Completed Metal doors and windows, Yerevan OTJ entrance doors and furniture 6 production Yerevan OTJ Tailoring service 10 Yerevan Yerevan OTJ Printing 5 Yerevan OTJ Massage 20 Yerevan OTJ Floristics 13 Yerevan OTJ Furniture production 5 Gyumri OTJ Computer operator 4 Shirak marz Gyumri OTJ Furniture production 10 Artik OTJ Floristics 3 Vanadzor OTJ Hairdressing 3 Vanadzor OTJ Silversmithing 7 Vanadzor OTJ AS Accounting training 12 Vanadzor OTJ Souvenirs production 8 Lori marz Vanadzor OTJ Manicure 6 Vanadzor OTJ Wedding accessories design 10 Alaverdi OTJ Hairdressing 4 Spitak OTJ Greenhouse keeping 10 Gegharkunik Metal-plastic doors and Chambarak OTJ 3 marz windows production Chambarak OTJ Hairdressing 3 Ararat marz Masis OTJ Hairdressing 9 Total 151 4. Sub-grant disbursement. From 92 pre-selected sub-grantees, the program ended up with 90 sub-grant contracts. All 90 sub-grant contracts have been fulfilled during the reporting period and the beneficiaries have launched their small businesses. The project achieved 125 percent of the preset indicators. 5. Sub-grants implementation monitoring and follow-up of established small businesses. Numerous visits have been conducted by a small-business expert and an M&E consultant to all the target locations for sub-grant contract signing, provision of on-the-spot consultancies, and monitoring of the launched businesses. Simultaneously, the small business expert has provided consultancies by phone. 27 Figure 3.4. Number of sub-grant signed contracts Target Location Sub-grantees’ Number of Number of Contracts Origin Location Sub-grantees Signed (businesses) Ararat marz Masis 3 3 Chambarak 8 8 Gegharkunik marz Gavar 3 3 Sevan 11 10 Abovyan 1 1 Kotayk marz Hrazdan 4 4 v. Nor Geghi 1 1 Alaverdi 2 2 Lori marz Spitak 6 6 Vanadzor 9 9 Shirak marz Gyumri 11 11 Yerevan Yerevan 33 32 Total 92 90 6. Information has been collected on the beneficiaries’ employment status (whether they are registered, formal employees or not) after completing the OTJ trainings, vocational trainings at educational centers, MA DRCs, and appropriate data are presented in Table 3.5. Figure 3.5. Number of employed beneficiaries by locations Number of Employed Number of Employed Target Location Beneficiaries Beneficiaries Total Registered Unregistered Yerevan Yerevan 25 20 45 Gyumri 7 4 11 Shirak marz Artik 1 0 1 Vanadzor 20 7 27 Lori marz Spitak 4 0 4 Alaverdi 4 0 4 Gavar 2 2 4 Gegharkunik marz Sevan 1 5 6 Chambarak 5 2 7 Kotayk marz Hrazdan 1 1 2 Total 70 41 111 Component 2: Provision of Vital Social Care and Services 7. Under Component 2, the MA delivered basic services to PWDs and other poor and vulnerable groups in their communities rather than at traditional large residential institutions. Component 2 of the project was designed to support the delivery of social services, such as counseling for about 4,000 beneficiaries, primary health services to about 3,000 of the beneficiaries, hot meals daily to about 2,050 of the beneficiaries, and also health care home services to about 500 bedridden beneficiaries. 28 Implementation of Activities 8. The MA applied its specific case management individualized model for the implementation of these services. This model offers integrated community-based services to the beneficiaries without segregating them from their homes and community, due to which the beneficiaries can have adequate and comprehensive access to quality services without changing their familiar and desirable environment. 9. The project supported different types of care based on each beneficiary’s individual needs. It also included social, psychological, socioeconomic, medical, and legal assistance; arrangement of technical preparedness trainings; activation of the beneficiaries’ potential resources; and increasing their self-esteem and self-reliance. The service cycle includes a needs assessment that helps identify the priorities for each specific beneficiary and the issues that need to be addressed. After compiling individual information and diagnosing the problem, individual case and intervention plans are developed. The individual intervention plan defines the problem addressing steps that have been agreed with the beneficiary. The social health services are provided by multidisciplinary teams (social workers, doctors, nurses, lawyer, care providers, and psychologists) based on the beneficiary needs assessment. 10. The MA provided the services through their infrastructures and home visits. The services are designed for both able-bodied and homebound beneficiaries. 11. During the project life, 4,856 beneficiaries per year had access to a variety of nutritional, social, and health services and 2,948 received hit meals. For the July–August 2016 period, a total of 3,161 beneficiaries had access to a variety of nutritional, social, and health services.  Up to 1,77113 beneficiaries had access to a hot meal five days a week and a total of 69,698 meals have been provided to the beneficiaries.  Totally, 3,161 benefited from social and health services. Out of these, 3,132 (788 males and 2,344 females) persons benefited from 20,999 social services and 2,779 (624 male and 2,155 female) persons had access to 14,864 health services.  About 349 persons had access to personal care services. 12. Tables 3.6 and 3.7 provide details of the total number of beneficiaries per location and per year over the life of the project. Figure 3.6. Total number of beneficiaries per province under Component 2 Number of Beneficiaries Number of Number Receiving Services Beneficiaries Province Number of Centers of MA (health, psychosocial) Receiving a Meal staff Male Female Male Female 13 The indicated number of beneficiaries also includes those beneficiaries whose service provision was terminated for some reasons, for example, because of illness, death, or absence from the country; residence address change; and so on during the reporting period. 29 Number of Beneficiaries Number of Number Receiving Services Beneficiaries Province Number of Centers of MA (health, psychosocial) Receiving a Meal staff Male Female Male Female Yerevan 6 (Shengavit, Avan- 2 548 1,678 452 720 Arinj, Nor-Nork, Davtashen, Kanaker- Zeytun, Ajapnyak) Ararat 2 (Artashat, Masis) 1 43 58 68 114 Kotayk 2 (Hrazdan, 1 158 377 120 174 Charencavan) Gegharku 3 (Gavar, Sevan, 3 133 310 119 227 nik Chambarak) Shirak 2 (Gyumri, Artik) 2 151 568 99 176 Lori 4 (Vanadzor, 2 275 557 244 435 Alaverdi, Spitak, Tashir) Total 19 11 1,308 3,548 1,102 1,846 Figure 3.7. Total number of beneficiaries per year under Component 2 Total Number of Beneficiaries 2013 2014 2015 2016 Male 812 1,072 988 905 Female 2,517 2,804 2,777 2,589 Total 3,329 3,876 3,765 3,494 Disabled 1,182 1,357 1,286 1,179 Poor 2,147 2,519 2,479 2,315 Total 3,329 3,876 3,765 3,494 Nutrition Services 13. For the project life, around 19 soup kitchens were operational in Yerevan and five target locations. These soup kitchens served beneficiaries every weekday. Through the support of volunteers, meals were delivered to those who could not visit the soup kitchen because of illness. The soup kitchens served the beneficiaries once per weekday. The MA prepared the menu in advance for every month and then forwarded it to all the operational soup kitchens. Thereby, all the soup kitchens had a standardized menu consisting of two to three courses, high-quality bread, and hot/cold beverages. 14. Food, vegetables, and bread for delivery to poor people at soup kitchens have been procured according to the World Bank procurement procedures. 15. The MA delivered social and health services to PWDs and other poor and vulnerable groups in their communities. The MA delivered these services at its multifunctional service centers, as well as through visiting the beneficiaries at their homes. The social and health services have been provided based on the need assessment, individual work, and intervention plans of each beneficiary. 30 Social and Psychological Services 16. The MA’s personnel mediated between the beneficiary and the Social Service Provision Department of the Government when necessary to ensure that eligible beneficiaries are provided with services. The MA’s personnel have also helped beneficiaries maintain personal hygiene at home through personal care services. These services include bathing and partial bathing, nail hygiene, and shaving; replacement of bed linen and underwear, laundry, ironing, pressing, and sewing; general cleaning of the apartment; and feeding. 17. Through its centers, the MA has helped beneficiaries resolve psychological issues and helped them cope with isolation, stress, and depression by conducting trainings and organizing different events. Health Services 18. Health services have been provided by the MA’s doctors and nurses through multifunctional service centers and at the homes of beneficiaries. Within the reporting period, 2,779 beneficiaries had access to a total of 14,864 health services. 19. At multifunctional service centers, the MA health personnel gave medical check-ups, counseled patients on their medical issues, and treated patients. The personnel prescribed medications directly to patients. The MA’s personnel made home visits to deliver health care to bedridden beneficiaries. Furthermore, the health personnel conducted trainings to disseminate information among the beneficiaries on healthy lifestyles, how to prevent seasonal outbreaks of disease, and on other health issues. Figure 3.8. Provided social health services by locations Community Social Services Health Services Marzes Center Number of Number of Number of Number of Location Beneficiaries Services Provided Beneficiaries Services Provided Nor Nork, Shengavit, Yerevan 1,436 10,516 1,195 6,690 Avan-Arinj Districts Shirak Gyumri, Artik 484 3,991 464 2,022 Vanadzor, Lori 516 2,934 468 1,951 Alaverdi, Spitak Sevan, Gavar, Gegharkunik 277 1,388 270 1,496 Chambarak Ararat Masis, Artashat 64 433 28 94 Hrazdan, Kotayk 355 1,737 354 2,611 Charentsavan Total 3,132 20,999 2,779 14,864 31 Component 3: Creating of an Inclusive Environment for People with Disabilities and Enhancement of their Self-advocacy 20. Component 3 helped PWDs become more independent and able to address and resolve some of their own difficulties. To achieve this, the component was designed to support better access to public facilities and to public information and advocacy education. Under this component, the following activities have been carried out: (a) Better access to public facilities and to public information (i) Establishment of DRCs for the dissemination of information to PWD and others (ii) Remodeling of public buildings so that PWDs can gain access; obtain information; and apply for documents, services, and benefits (b) Advocacy education (i) Skills training on advocacy and self-help (ii) Creation of self-help groups 21. During the reporting period, rehabilitation and remodeling activities have been completed at Hrazdan and Shengavit DRCs. 22. Trainings have been conducted on the following topics:  Advocacy tools  State regulatory programs of employment implemented by the employment agency  Privileges and rights of PWDs: frequently asked questions  Labor rights of disabled people in Armenia  Family benefits system of the Republic of Armenia  Family-oriented social welfare policy of the Republic of Armenia  Social security system in Armenia  The concepts of poverty  Social welfare system of the Republic of Armenia: goals and functions 23. Self-help groups have been organized by social work consultants consisting of poor and disabled people, who need to participate in their community’s active life and gain new capacities and skills to solve problems independently. Then advocacy strategies have been developed by social work consultants in cooperation with self-help groups’ members under the supervision of a trainer. The strategies have been developed in the following directions: (a) Employment promotion of PWDs (b) Participation promotion in local self-government (community council meetings) (c) Promotion of community environmental problems’ solving (d) Promotion of local transport launch (e) Promotion of health services’ accessibility in community 32 24. Table 3.9 provides the total number of beneficiaries over the life of the project under Component 3. Figure 3.9. Total number of beneficiaries receiving advocacy and self-help training Number of Beneficiaries 2014 2015 2016 Male 141 258 148 Female 380 770 453 Total 521 1,028 601 Of which PWDs 304 522 225 Of which poor 217 506 376 25. During the project period, 12 self-help groups continued their activities and most of them reached the goals mentioned in their advocacy strategies. Figure 3.10. Total number of the conducted skills trainings on advocacy and self-help by locations Number of Number of Trained Target Location Trainings Beneficiaries Yerevan 2 40 Shirak 4 32 Lori 4 45 Gegharkunik 4 44 Ararat 0 0 Kotayk 2 17 Total 16 178 33 34