No. 0402 Social Protection Discussion Paper Series Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries Gordon Betcherman, Karina Olivas and Amit Dar January 2004 Social Protection Unit Human Development Network The World Bank Social Protection Discussion Papers are not formal publications of the World Bank. They present preliminary and unpolished results of analysis that are circulated to encourage discussion and comment; citation and the use of such a paper should take account of its provisional character. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. For free copies of this paper, please contact the Social Protection Advisory Service, The World Bank, 1818 H Street, N.W., Washington, D.C. 20433 USA. Telephone: (202) 458-5267, Fax: (202) 614-0471, E-mail: socialprotection@worldbank.org. Or visit the Social Protection website at http://www.worldbank.org/sp. Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries Gordon Betcherman, Karina Olivas, and Amit Dar January 2004 Summary Active labor market programs (ALMPs) are used to reduce the risk of unemployment and to increase the earnings capacity of workers. Particular interventions include employment services, training, public works, wage and employment subsidies, and self-employment assistance. These programs are implemented to enhance labor supply (e.g., training); increase labor demand (e.g., public works, subsidies); and improve the functioning of the labor market (e.g., employment services). ALMPs are often targeted to the long-term unemployed, workers in poor families, and particular groups with labor market disadvantages. These programs have important social, as well as economic, objectives. OECD countries, in particular, have a long and extensive experience with ALMPs. These programs are becoming more relevant in developing and transition countries, too, as many governments grapple with growing unemployment and underemployment problems. How much reliance should countries place on active labor market programs? This is a controversial question. Proponents argue that they are the most direct instrument for dealing with unemployment and poverty among workers. Opponents counter that ALMPs are largely a waste of public funds and that any observed benefits for participants are usually at the expense of other workers. It is important then, to rigorously evaluate the impacts of these programs and their cost-effectiveness. Earlier reviews of impact evaluations by the World Bank, the OECD, and others have concluded that policy-makers must be cautious regarding what ALMPs can realistically achieve. The evidence suggested that these programs were not a panacea for unemployment but some types of interventions, properly designed, could be effective for some workers. It should be noted that these reviews were based almost exclusively on the experience of industrialized countries because very few evaluations existed anywhere else. In this paper, we build on the 72 scientific (i.e., control-group) evaluations considered in the previous World Bank study (Dar and Tzannatos, 1999) by adding 87 new studies. With this additional evidence, we can ask whether the original conclusions still hold. Moreover, with some evaluations now being carried out in transition and, to a lesser extent, developing countries, we can begin to ask whether the findings of impact evaluations in industrialized countries apply in these contexts as well. This updated review does not change the overall findings from the 1999 study on the impacts of ALMPs in any fundamental way. A wide range of results can still be found with some programs demonstrating positive labor market effects for participants and others showing either no impact or even negative effects. Obviously, program design and the context in which the program operates matters a great deal. While it is increasingly difficult to isolate impacts of particular types of programs because of a trend to integrated service provision, our review leads to the following general conclusions: i · Employment services. These services include counseling, placement assistance, job matching, labor exchanges, and other related services. They generally have positive impacts on the post-program employment and earnings of participants. Costs are relatively low so the cost-benefit ratio is often favorable. However, employment services ­ at least by themselves ­ are of limited use in situations where structural unemployment is high and there is a lack of demand for labor. There are some questions about the coverage and effectiveness of these services in developing countries where many labor market transactions are informal. · Training for the unemployed. Participants often benefit from these programs in terms of higher employment rates but not in terms of higher earnings. The few evaluations in developing countries paint a less favorable picture. Programs seem to work best with on- the-job training and active employer involvement. Results are more positive for women than men. · Retraining for workers in mass layoffs. These programs most often have no positive impacts, although there are exceptions. The few successful cases typically include a comprehensive package of employment services to accompany the retraining. However, these are generally expensive. · Training for youth. These programs are almost always unsuccessful in improving labor market outcomes, at least in developed countries. It makes much more sense to invest earlier in the education system to reduce drop-outs and other schooling problems. While there are few studies in developing countries, evaluations in Latin America do find positive impacts for programs that integrate training with remedial education, job search assistance, and social services. · Wage/employment subsidies. Most often these do not have a positive impact and have substantial deadweight and substitution costs. Targeting and monitoring may help but at the cost of reducing take-up rates. · Public works. This can be an effective short-term safety net but public works do not improve future labor market prospects for participants. · Micro-enterprise development/self-employment assistance. There is some evidence of positive impacts for older and better-educated workers. However, take-up is low. What can we say about the impacts of ALMPs in developing and transition countries in particular? Many findings from industrialized countries do seem to apply broadly to transition countries but ­ on the basis of what is still a small sample of studies -- this is not always true in the case of developing countries. The much larger informal labor markets and weaker capacity to implement programs may limit what some programs can achieve in terms of creating formal employment or increasing wages. The few evaluations in these countries for employment services and training programs for the unemployed are less positive than the (much larger) body of evidence in the OECD and transition countries. On the other hand, some youth training programs in developing countries have much more positive impacts than are seen in industrialized countries. It may be that such programs in these low-income labor markets have more potential because abundant supplies of skilled workers are not available. The sample of evaluations outside the OECD is still limited, especially in low-income developing countries, and further studies will be needed to confirm these initial observations. ii At any rate, the ingredients for successful interventions seem to apply for all countries. Comprehensive packages of services, programs that are oriented to labor demand and linked to real workplaces, and careful targeting are good design features. Finally, the evaluations underline the fact that program impacts are usually more positive when the economy is growing. While our knowledge on the impacts of ALMPs continues to grow, there is still much more to learn, especially in the context of developing and transition countries. Evaluations rarely track post-program outcomes beyond a couple of years so little evidence exists on longer- term impacts. Many studies do not estimate the deadweight, substitution, and displacement effects and thus cannot account for the general equilibrium impacts of programs. Many do not fully consider program costs and, as a consequence, cannot inform on the key policy issue of efficiency. Finally, the evaluation literature provides insights into what works but far less on why. Ultimately, policy-makers need to understand what circumstances and design features explain effective outcomes for specific groups. Despite the mixed evaluation picture, governments have little choice but to use active programming as one instrument in their response to the economic and social problems associated with unemployment and poverty in the labor force. They should be realistic about what ALMPs can achieve and allocate resources on the basis of cost-effectiveness. The challenge, then, is to learn from existing experiences, investing in programs that have positive returns and altering or dropping programs that do not. If ALMPs are going to be an economically useful policy, it is very important that governments carefully evaluate their own programs and introduce interventions on the basis of what works domestically and in other countries. iii Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries Gordon Betcherman, Karina Olivas and Amit Dar1 1. INTRODUCTION In this paper, we provide an overview of the recent international experience with active labor market programs (ALMPs). Basing our evidence on the growing body of program evaluations, we focus on the impacts of ALMPs on the subsequent employment and earnings of participants. This paper provides an update to earlier assessments (e.g., Fay, 1996; Dar and Tzannatos, 1999; Martin, 2000; Martin and Grubb, 2001) by incorporating the results of the more recent program evaluations. It also extends these previous reviews by explicitly considering the impacts of ALMPs in developing and transition countries. While most rigorous program evaluations continue to be undertaken in industrialized countries, for the first time we now have a significant number of evaluations from transition and, to a lesser extent, developing countries. Active labor market programs, as defined in this review, include employment services, training and retraining, public works, wage and employment subsidies, and self- employment assistance. These programs are implemented to enhance labor supply (e.g., training); increase labor demand (e.g., public works, subsidies); and improve the functioning of the labor market (e.g., employment services). Their objective is primarily economic ­ most often, to increase the probability that the unemployed will find jobs or that the underemployed will increase their productivity and earnings. It is the economic aspect that provides the focus for most evaluations and, indeed, for this review. However, while not explicitly addressed in the evaluations reviewed in this paper, the social and political dimensions of ALMPs should also be recognized. These programs can contribute to social inclusion and cohesion by increasing productive employment. They also can have political 1 Authors are: Gordon Betcherman, Social Protection Unit, World Bank (Gbetcherman@worldbank.org); Karina Olivas, Consultant to the World Bank (olivas_kari@yahoo.com); and Amit Dar, South Asia Region, World Bank (Adar@worldbank.org). This paper has benefited from comments on an earlier draft provided by David Fretwell, Martin Godfrey, and the participants in a World Bank seminar held on October 6, 2003. 1 benefits by visibly demonstrating that governments are responding actively to labor market problems, such as unemployment or poverty in the workforce. OECD countries, in particular, have a long and extensive experience with ALMPs. These programs are becoming more relevant in developing and transition countries, too, as many governments grapple with growing unemployment and underemployment. Past reviews of evaluation studies, however, have raised questions about the real impacts and efficiency of these programs. The previous assessment of evaluation results by the World Bank concluded that "... payoffs are usually modest and cost-benefit analysis suggests that social rates of return to effective programs are also sometimes negative." (Dar and Tzannatos, 1999). An OECD review drew a similar conclusion: "[t]he track record of many active measures is mixed in terms of raising future employment and earnings prospects of job seekers and producing benefits to society." (Martin, 2000). The findings of these reviews imply that ALMPs are not a panacea for large-scale unemployment and that expectations must be realistic. They also point to the importance of learning from past evidence on what works in terms of design, targeting, and implementation. With new impact evaluations coming out every year, it is important to reassess the conclusions drawn from the earlier reviews which covered studies up to the mid-to-late 1990s. In this paper, we build on the 72 scientific evaluations considered in the previous World Bank study (Dar and Tzannatos, 1999) by adding 87 new studies.2 With this additional evidence, we can ask whether the original conclusions still hold. We can also now begin to address the question of whether the findings of impact evaluations in OECD countries also hold for developing and transition countries. The original study was essentially limited to the developed-country experience because 90% of the available studies at that time were from these countries. However, 39 of the 87 new evaluations investigate the impacts of programs in developing and transition countries. The extended sample of evaluations does not change the findings from the 1999 study on the impacts of ALMPs in any fundamental way. A wide range of results can still be found with some programs demonstrating positive labor market effects for participants and others 2Of the evaluations included in the original review, we have included only those that used a control-group methodology. This was also the criterion for including new evaluations. 2 showing either no impact or even negative effects. The most effective type of intervention continues to be employment services which, according to most evaluations, can improve future employment and earnings prospects in a cost-effective manner. However, employment services ­ at least by themselves ­ are of limited use in situations where structural unemployment is high and there is a lack of demand for labor. The picture on training and retraining is mixed: a somewhat positive assessment emerges for programs targeted to the long-term unemployed but less so for retraining in mass layoff cases and training for unemployed or disadvantaged youth. Most evaluations of wage and employment subsidies and public works demonstrate no positive impacts for participants in terms of post-program employment or earnings. The few evaluations that look at micro-enterprise development/self- employment assistance offer a mixed picture of whether these programs help future earnings. It appears that these patterns apply broadly across developed, transition, and (with far less confidence) developing countries. There are a few variations ­ for example, training programs for youth seem to have a more positive impact in developing countries and public works evaluations are more favorable in transition countries than in developed countries. However, these observations are very tentative given that the inventory of evaluations outside the OECD remains quite small. While our knowledge on the impacts of ALMPs continues to grow, there is still much more to learn, especially in the context of developing and transition countries. Evaluations usually do not track post-program outcomes beyond a couple of years so little evidence exists on longer-term impacts. Many studies do not estimate the deadweight, substitution, and displacement effects and thus cannot account for the general equilibrium impacts of programs. Many do not fully consider program costs and, as a consequence, cannot inform on the key policy issue of efficiency. Finally, the evaluation literature provides insights into what works but far less on why. Ultimately, policy-makers need to understand what circumstances and design features explain effective outcomes for specific groups. This clearly needs to be a focus of future research. 3 The next section provides background on active labor market programs. Section 3 covers key elements of evaluation methodology. In section 4, we turn to the evaluations themselves, reviewing the body of studies that is now available. Finally, conclusions are drawn in section 5. 2. AN OVERVIEW OF ACTIVE LABOR MARKET PROGRAMS In this section, we present an overview of active labor market programs including a brief description, objectives, and recent trends including spending on ALMPs. 2.1 What are Active Labor Market Programs? The following types of programs are considered as ALMPs: · Employment Services. These services fulfill brokerage functions, matching available jobs with job seekers. This assistance comprises many different types of activities including initial interviews at employment offices, in-depth counseling during an unemployment spell, job clubs, labor exchanges, etc. Traditionally, employment services were provided exclusively by public agencies but now, in many countries, public and private services coexist, usually serving different clientele. Often, public employment services target the disadvantaged and the long-term unemployed while private agencies focus more on the employed, skilled, and white-collar workers. These services are relatively inexpensive compared to other ALMPs and they can help in shortening unemployment spells. On the negative side, these interventions can have "deadweight losses" ­ i.e., individuals who find jobs through these services are often more qualified than most job-seekers and many perhaps could have found jobs in the absence of these services. · Labor Market Training. This includes training that is publicly-supported, usually through either direct provision (e.g., through public training institutes) or financial support (e.g., funding training costs and/or subsidizing trainees). In many countries, governments are focusing on addressing market failures in information and financing, while leaving more of the delivery to private providers. Training programs can be concerned with developing basic job readiness or have specific vocational skill content. 4 They may be comprehensive in terms of their coverage or target specific groups such as the long term-unemployed, workers displaced in mass layoffs, or young people, often with special attention to school drop-outs. While training programs can lead to increases in productivity and employability, they can be costly and of limited use when job opportunities for trained workers are scarce. · Job Creation. These programs are intended to support the creation of new jobs or the maintenance of existing ones. Three types of programs fall under this category. (1) Wage/employment subsidies. These are subsidies to encourage employers to hire new workers or to keep employees who might otherwise have been laid off for business reasons. They usually take the form of direct wage subsidies (directed to either the employer or worker) or social security payment offsets. These programs typically are targeted to the long-term unemployed, areas/sectors with high unemployment, and special groups of workers (e.g., youth). While these programs serve a social objective, design is critical in order for subsidies to create jobs in a cost-effective manner. They are often associated with deadweight losses. They also can have unintended effects such as subsidized workers replacing unsubsidized ones ("substitution" effect) or employers hiring subsidized workers and laying them off once the subsidy period ends. (2) Public works. These programs -- known by a range of terms including temporary community projects, labor-intensive projects, and workfare -- involve direct job creation through public works or other activities that produce public goods or services. They can alleviate unemployment or short-term poverty by creating temporary jobs and can help disadvantaged, poor, and long-term unemployed workers to regain contact with the labor market. Governments can manage these projects directly or contract with non-profit organizations or private businesses. On the positive side, these programs can lead to the production of public goods/services and develop basic physical or social infrastructure; indeed, in many cases, this, rather than job creation, is the main objective. These programs can also be effective short-term safety nets. On the negative side, it is often observed that the long-term labor market 5 impact of these programs is often insignificant, and, in some countries, there is a stigma attached to public works jobs which may decrease the employability of participants over the long run. (3) Micro-enterprise development/self-employment assistance. These programs offer assistance to unemployed workers to start their own enterprises. This can involve providing financial and advisory support for start-up, "incubator" services, or supporting operating costs of small businesses. These programs have been offered both on a universal basis or to particular groups, such as the newly unemployed or the long-term unemployed. In some cases, financial contributions are made through an initial lump-sum payment and, in other cases, they involve periodic allowances. Often there is "screening" whereby potential beneficiaries undergo an assessment which evaluates their likelihood of success. Public programs to support small business loans can contribute to the removal of distortions arising from credit rationing. However, most unemployed are looking for jobs rather than entrepreneurial opportunities. These programs also risk placing small businesses that do not get assistance at a disadvantage relative to those that do. While this list captures the main categories of ALMPs, interventions are often combined in individual programs. 2.2 Objectives of ALMPs ALMPs include economic, social, and political objectives. The focus in this paper is on economic objectives and the evaluations included in our review are concerned with the labor market impacts. However, these programs almost always have social and political objectives that can be very important to the government that implements them. They signal concern with unemployment and workforce poverty and represent a tangible policy that directly is meant to address these problems. When targeted on specific groups such as youth or the disabled, they can demonstrate the government's priority on issues of equity and access for the vulnerable. ALMPs can also play an important part in reducing resistance to restructuring. 6 The principal economic objective of active labor market programs is to increase employment and incomes. This can be achieved in various ways. ALMPs can play a stabilization role in the sense of governments directly providing temporary jobs through public works or by shifting labor supply or demand curves outward by offering training or wage subsidies. Training, mobility incentives, and other employment services such as job search assistance can reduce structural imbalances by improving the match between workers and jobs. By decreasing the number of vacancies at a given level of unemployment, ALMPs can also increase employment by reducing both upward wage pressures and labor bottlenecks. Employment and income effects of active programs can also be transmitted through increases in skills and productivity. Even where net employment effects may not be significant for some active labor market programs, they can increase the attachment of the long-term unemployed to the labor force and decrease their dependence on unemployment benefits or other social assistance. By assisting the most disadvantaged workers, ALMPs can break down potentially negative consequences associated with "outsider" phenomena. Active programs, then, can serve various objectives and policy-makers need to be clear about which are their priorities. The economic orientation of an ALMP strategy can be to moderate cyclical downturns, reduce structural imbalances or otherwise improve the functioning of the labor market, increase productivity, support disadvantaged or at-risk workers, support at-risk employers or industries, or some combination of the above. Each of these objectives calls for different types of ALMPs and different client populations. Table 1 provides an illustration of how policies might differ depending on objectives. 2.3 Trends in Active Labor Market Programs Most empirical evidence on the level and composition of active labor market programs pertains to developed countries. More recently, there have been some data available for transition countries but virtually nothing systematic for developing countries. 7 Table 1: Tailoring Programs to Objectives Objective Program orientation Targeting orientation Moderate cyclical · Direct job creation (e.g., public works) · Vulnerable groups (with downturns · Wage subsidies least resiliency) · Training (subsidies or grants to workers or employers) · Hard-hit regions and · Self-employment support industries Reduce structural · Employment services (e.g., information, search · Proximate regions, imbalances assistance, mobility assistance) industries, or · Training occupations · Wage subsidies Improve general · Employment services · All labor market · Training (e.g., apprenticeship, school to work functioning transition) Enhance skills and · Training and retraining (including in-service, · At risk or disadvantaged productivity apprenticeship) worker categories (especially for retraining) Support · Employment services (counseling, job search · At-risk or disadvantaged disadvantaged or at- assistance) worker categories risk workers · Training (e.g., grants, subsidies) · Wage subsidies Source: Betcherman, Dar, Luinstra, and Ogawa (2000) 2.3.1 Industrialized Countries The OECD has collected statistics on ALMP expenditures by member countries since the mid-1980s. These expenditures have also been compared with spending on passive programs (e.g., unemployment insurance, unemployment assistance, etc.) to assess trends in the balance between the two. Chart 1 presents relative spending on active and passive programs among OECD countries from 1990-2002. Throughout this period, average national expenditures on active programs have remained relatively constant at around 0.75% of GDP, peaking temporarily during the 1992-95 downturn at close to 1%. In comparison, spending on passive programs has been higher and has fluctuated more, with a significant rise during the early 1990s recession to nearly 2% of GDP.3 In all OECD countries except Italy, Norway, Portugal, and Sweden, spending on active measures is less than spending on passive programs. 3 In such periods, while spending on both active and passive programs increases, the rate of increase for the latter is generally much greater because spending on passive programs like UI is largely non-discretionary (i.e., plans have benefit obligations that have to be met). 8 Chart 1: Public Expenditures on Active and Passive Programs in OECD Countries, 1990-2002 Public expenditure in Active and Passive Measures in Labor Market in OECD countries 1990-2002 (percentage of GDP) 2.00 Active Measures 1.75 Passive 1.50 M 1.25 1.00 0.75 0.50 0.25 0.00 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Source: OECD (2003) In terms of the overall composition of ALMP spending in OECD countries, training represented the largest category, 36% of total expenditures, in 2001-02 (Table 2). Public employment services and job subsidies accounted for 25% and 20%, respectively. Note that countries generally spend only a very small share on micro-enterprise development. These aggregate trends on ALMP spending mask important differences across developed countries in terms of both level and composition of ALMP expenditures. As Table 2 indicates, some European countries (e.g., Netherlands, Denmark, France, Belgium, Germany, Sweden) spend well over 1% of GDP while the United States, Japan, Korea, and the United Kingdom spend under 0.4%. The table also illustrates considerable variation in how ALMP funds are allocated across program types. For example, many countries with small overall ALMP programs (e.g., Japan and English-speaking countries) tend to place relative emphasis on employment services. On the other hand, the share of national spending on training tends to be greatest in European countries with activist orientations (e.g., Denmark, France). Country spending on type of program as a percentage of GDP is shown in a series of charts in Annex 1. 9 Table 2: Distribution of Expenditures on ALMPs in Industrialized OECD Countries, 2001-02 Public Training Job Micro- Others Total as Employment Subsidies Enterprises (incl. % of Service disabled) GDP % distribution Australia 44.4 20.0 17.8 4.4 13.4 0.45 Austria 26.4 43.4 17.0 3.8 9.4 0.53 Belgium 15.6 20.3 53.9 0.0 10.2 1.28 Canada 46.5 39.5 4.7 2.3 7.0 0.43 Denmark 7.6 60.8 10.8 0.0 20.8 1.58 Finland 11.9 46.5 29.7 3.0 8.9 1.01 France 13.8 51.5 26.9 0.0 7.8 1.30 Germany 19.2 35.0 14.2 4.2 27.4 1.20 Japan 60.7 10.7 25.0 0.0 3.6 0.28 Korea 17.9 35.7 37.7 3.6 5.1 0.28 Netherlands 14.9 29.3 21.8 0.0 55.6 1.74 New Zealand 24.1 51.9 9.3 5.6 9.1 0.54 Norway 15.1 7.0 1.2 0.0 76.7 0.86 Portugal 18.0 60.7 9.8 4.9 6.6 0.61 Spain 10.6 31.8 45.9 5.9 5.8 0.85 Sweden 27.0 22.0 12.1 2.8 36.1 1.41 Switzerland 22.2 25.9 18.5 1.9 31.5 0.54 United Kingdom 42.1 42.1 7.9 0.0 7.9 0.38 United States 26.7 46.7 6.6 0.0 20.0 0.15 Unweighted Average 24.5 35.8 19.5 2.5 19.1 0.81 Source: OECD (2003) While overall spending on ALMPs in industrialized countries has been relatively constant, the strategy, design, and implementation of active policy has undergone some significant changes over the past decade. Specifics differ by country but in most industrialized countries, many of the following trends are evident: · Increased emphasis on job search assistance. This assistance often has been linked with closer monitoring and tighter job search requirements for unemployment benefit recipients. · Integrated services (one-stop window/guichet unique) so that clients can receive information, counseling, and access to services from a single source. · An increasing reliance on private delivery of services (e.g., for training, employment services, and public works). Government's role in these situations has been to establish overall priorities, ensure quality, and provide financing, especially to address equity concerns. 10 · Closer partnership with employers and communities in order to ensure that programs respond to market conditions. · "Making work pay" initiatives. These have used employment subsidies and tax benefits primarily to encourage low-skill workers to take low-pay jobs. · "Profiling" of unemployed clients to identify those likely to need employment services or retraining. In some countries, program participation (based on profiling) is an obligatory condition to qualify for unemployment benefits. · Increased emphasis on program evaluation and allocation of resources on the basis of evaluation results. 2.3.2 Transition and Developing Countries As noted above, some evidence is now accumulating for the transition countries but this does not have the coverage over time nor the standardization available for the industrialized OECD countries. O'Leary, Nesperova, and Samorodov (2001) have assembled expenditure figures for 9 transition countries, based on 1998 data from national employment services. These figures form the basis for Table 3. While these percentages are not identical to the OECD database, they do appear to be generally comparable.4 The data show that the transition countries rank toward the lower end of ALMP spending (as a share of GDP) when compared to industrialized countries. One group of countries (Czech Republic, Croatia, Ukraine, Estonia, and Russia) spends less than .15% of GDP, below all industrialized countries while the other (Slovakia, Poland, Hungary, and Bulgaria) spends between .34% and .54% of GDP, which is above the lowest-spending industrialized countries, but below the OECD average. In many transition countries, most active spending is allocated to the employment service. However, this is not true of all countries. For example, according to the 1998 figures, training accounts for over one-half of the expenditures in Estonia, while direct job creation (public works and subsidized employment) is the major intervention in Poland and Slovakia (O'Leary, Nesperova, and Samorodov, 2001). In terms of the balance between active and passive spending, the transition countries are very similar to industrialized ones. 4 There are some transition countries also now included in the OECD database on which to base this comparison. The OECD's latest calculations for 2002 indicate that the Czech Republic spends 0.21% of GDP on ALMPs and Hungary spent 0.47%. The figures reported by O'Leary et al. (2001) for 1998 are 0.14% and 0.39%, respectively. 11 Active spending was 61% of passive spending for the nine countries in Table 3, while the corresponding figure for the OECD countries was 65%. Table 3: Expenditures on Active and Passive Labor Market Programs in Selected Transition Countries, 1998 Active Spending as % Passive Spending as Total Spending as % of GDP % of GDP of GDP Bulgaria 0.12 0.46 0.8 Croatia 0.03 0.48 0.6 Czech 0.05 0.26 0.4 Estonia 0.07 0.10 0.2 Hungary 0.28 0.91 1.3 Poland 0.30 0.59 1.0 Russia 0.02 0.13 0.2 Slovakia 0.32 0.56 1.1 Ukraine 0.03 0.19 0.3 Unweighted average 0.25 0.41 0.66 Source: Based on O'Leary, Nesperova, and Samorodov (2001), Table 2.4 Systematic statistical overviews of ALMP activities in developing countries are not available. The evaluation studies that we have found, however, suggest that Latin American countries have been investing significantly in recent years in youth employment training and public works programs. A review of ALMPs in East Asia concluded that public works have been implemented fairly widely but other forms of active programs have not been used on any significant scale (Betcherman, Dar, Luinstra, and Ogawa, 2000). Little evidence is available for North Africa and the Middle East although a recent detailed review of Tunisia showed that self-employment assistance (largely through micro-financing) and youth training dominated the ALMP profile in that country (World Bank, 2003). In Africa, there is very little active programming on any significant scale. 3. EVALUATION METHODOLOGY: ISSUES AND KEY CONCEPTS There are different types of program evaluations including process evaluations that focus on how well a program is delivered and performance monitoring which is concerned with timely indicators of how well program objectives are being achieved. These are very important tools for improving program effectiveness and can be carried out even in countries with low administrative and analytical capacity. However, the incremental value of a program and its cost effectiveness can only be calculated through impact evaluations. These 12 impact evaluations, when conducted rigorously, can identify the effects of a given program on participants, and when coupled with cost information, can reveal the net benefits of programs to participants, to government finances, and in some cases to the broader labor market and society. However, reliable impact evaluations require careful methodological design, good data, and, for some methods, sophisticated econometric techniques. Since there is an extensive literature on evaluation methodology, we only briefly cover some important aspects in this section.5 Generally, the central questions for ALMP impact evaluations are: (1) what are the impacts of program participation on the future labor market situation of participants? and, (2) What is the cost-effectiveness of programs? Unfortunately most evaluations focus on the first question only, with a very few adequately addressing the cost question. In most ALMP evaluations, the key impact indicators are post-program employment rates and earnings. However, some evaluations do look at other social indicators such as criminal rates, teenage pregnancy, government benefit recipiency, etc. 3.1 Methodological Approaches Whatever the outcomes of interest, the real net impacts of a program cannot be answered by simply tracking the post-program experiences of participants. This approach will not reveal anything about what would have happened to these workers if they had not taken the program (i.e., the "counterfactual"). Since it is not possible to observe this counterfactual directly, a scientific evaluation must approximate what would have happened to workers had they not participated. This is done by constructing a control group of people who did not participate in the program but ideally are identical to participants in all other ways. The outcomes (e.g., employment, earnings) of the two groups are then compared at some point after the program has been completed in order to assess the impacts of the program. A control group is absolutely essential for a valid impact evaluation. Construction of the control group is usually the most important challenge in designing an impact 5 See, for example, Heckman, LaLonde, and Smith (1999); Benus and Orr (2000); Grubb and Ryan (1999); O'Leary, Nesperova, and Samorodov (2001); and Kluve and Schmidt (2002). A practical summary is available in World Bank (2002a). 13 evaluation methodology. This challenge is complicated by the fact that some relevant characteristics of individuals may be difficult to observe (e.g., ambition, drive, etc.). Techniques using control groups are of two types: experimental and quasi- experimental. Experimental evaluations require selection of treatment and control groups prior to the intervention. In quasi-experimental studies, treatment and control groups can be selected before, during, or after the intervention. Experimental techniques. These are based on the principle that when individuals are randomly assigned to treatment and control groups, observable and unobservable characteristics of the two groups should not differ on average, and so any difference in outcomes can be attributed to program participation. It is generally agreed that properly conducted randomized experiments embody the best control-group properties for producing consistent and non-biased estimates of program impact. Furthermore, the results are simple to interpret -- the program impact is the difference in the means of the variable of interest between the sample of program participants and control group. For example, if the mean post-program employment rate for participants in a training program is 60% and the rate is 50% for non-participants, then the net program impact is 10%. However, circumstances may dictate that a randomized experiment cannot be implemented properly or at all. Experiments can be costly, may be disruptive to the operation of the program, and can pose ethical questions about excluding people from the intervention. In addition there can be technical problems -- for example, assignment may not be truly random or there may be group "contamination" (e.g., control group members receiving alternative but similar services). Quasi-experimental techniques. When experiments are not feasible or are not undertaken for some other reason, evaluators must construct a control group through non- experimental selection. This is typically done by drawing from a survey or administrative database that includes individuals with characteristics that fit the targets of the intervention. In order to get unbiased estimates of program impact, any differences in the characteristics of the control and treatment groups that might affect the outcome of interest must be accounted for using econometric techniques. This can be technically difficult for a variety of reasons including the likelihood that individuals may have unobservable characteristics that can 14 affect program outcomes or even participation. Many techniques, often highly sophisticated, have been developed to address the challenge of generating unbiased program impact estimates through quasi-experimental methods. These include matching techniques that mimic an experimental control group, and selection-corrected and difference-in-difference techniques that attempt to control for unobservable characteristics affecting program participation and outcomes. The main attributes of quasi-experimental evaluations are that they can use existing data sources and can be relatively low cost, they can be undertaken in situations where experiments are problematic, and they can be done at any time after the program has begun. However, there are disadvantages. The most important is that, highly sophisticated methods notwithstanding, quasi-experimental evaluations do not always result in unbiased impact estimates.6 Estimates are often highly sensitive to model specification. Box 2 provides an example of this point. Finally, the complex techniques that are usually required make it difficult for non-specialists (including policy-makers) to interpret results. The attention paid to methodology in the literature should not obscure the importance of good data. Valid impact evaluations require both (Heckman et al., 1999). As Smith (2000) emphasizes, governments can do many things to improve the data available for evaluations including making data from both administrative and survey sources widely available (with appropriate privacy precautions) to the research community. 3.2 General Equilibrium Effects A complete assessment of the impact of a program requires an analysis of the program's general equilibrium effects ­ i.e., overall effects on the labor market because of indirect effects on people who are not participants. Partial equilibrium evaluations that focus only on the direct impacts of programs on participants cannot capture these overall effects. For example, an evaluation may overstate the impact of a program if it has not taken into account the possibility that some employment gains might have occurred even without the program or that observed gains have been at the expense of non-participants. These are the 6The validity of this approach has received considerable questioning since LaLonde (1986) showed that non- experimental results differed significantly from results of a randomized experiment. 15 deadweight, substitution, and displacement effects described in Box 1. Another potential effect involves the labor market impacts of taxes required to fund a program. Box 1: Some Commonly Used Terms in the Impact Evaluation Literature Additionality: This is the net increase in jobs created. It is the total number of subsidized jobs less deadweight, substitution and displacement effects. Deadweight Loss: Program outcomes are no different from what would have happened in the absence of the program. For example, wage subsidies place a worker in a firm that would have hired the worker in the absence of the subsidy. Displacement Effect: This usually refers to displacement in the product market. A firm with subsidized workers increases output but displaces output among firms without subsidized workers. Randomization Bias: This refers to bias in random-assignment experiments. In essence, this says that the behavior of individuals in an experiment will be different because of the experiment itself and not because of the goal of the experiment. Individuals in an experiment know that they are part of a treatment group and may act differently, as could individuals in the control group. Selection Bias: Program outcomes are influenced by unobservables not controlled for in an evaluation process (e.g. individual ability). Such factors can arise as a by-product of the selection process into programs where individuals "most likely to succeed" are selected into the program. Substitution Effect: A worker hired in a subsidized job is substituted for an unsubsidized worker who otherwise would have been hired. The net employment effect is thus zero. Treatment and Control Group: Program beneficiaries are the "treatment" group. In a scientific evaluation, their outcomes are compared with a "control" group of non-participants. Source: World Bank (2002a) Box 2: Impact Estimates for Participation in Retraining Programs, Hungary Quasi-experimental techniques were used to analyze the impact of training for 1992 graduates of Hungarian training institutions. Using different methodologies, significantly different estimates of the impact were computed. Estimation methodology Gain in employment Earnings gain ($/month) probability (%) Simple difference in means 19.2* 14.9* Quasi-experimental techniques Matched pairs 1.2 20.5 Correcting for observables 6.3* 4.9 Correcting for observables and 32.0* N/a non-observables * indicates statistically significant estimate. On trying different specifications, the evaluators concluded that the high estimates obtained using the correcting-for-unobservables technique were extremely sensitive to the empirical specification used. They felt that these estimates were unreliable and that the true employment impact of the program lay between the 1.2% and 6.3% generated by the matched-pairs and the correcting-for-observables techniques respectively. Source: Based on World Bank (2002a) 16 Not all ALMPs are likely to have general equilibrium effects -- for example, small- scale interventions or those that have no partial equilibrium impacts. But even where general equilibrium effects are quite likely, evaluations often do not try to estimate them because of the technical and data difficulties involved (Kluve and Schmidt, 2002). As Heckman et al. (1999) and Smith (2000) point out, accounting for these effects can require general equilibrium models of the labor market and the value and feasibility of these models relative to the more traditional evaluation methods remain an open question. 3.3 Cost-Benefit Analysis Most evaluations focus on whether a program improves the subsequent labor market situation of participants. If these impacts are not quantified monetarily and then related to costs, however, policy-makers do not have all of the necessary information to decide whether the program is justified on efficiency grounds. A program may be effective in the sense of creating benefits for participants (e.g., through employment or higher earnings) but may not be worthwhile from a social perspective if the benefits are less than the costs involved. Actually quantifying the costs of a program can be a complicated activity, involving gathering data from many sources including administrative databases. The major cost components in a typical ALMP evaluation will include the costs of administering the program, delivering the services, and the participants' opportunity costs. Cost-benefit analysis can be carried out from different perspectives including participants, government, and society as a whole. Unfortunately, most evaluations do not include rigorous cost-benefit analysis (Heckman et al., 1999). 4. EVALUATION RESULTS We now turn to the latest evidence from program evaluations on the impacts of ALMPs. This review updates and extends the previous World Bank study (Dar and Tzannatos, 1999) in two main ways. First, it expands the sample of studies we can use to assess overall ALMP impacts. We have been able to find 87 new impact evaluation studies that we have added to the evaluations included in the earlier review. Second, we can consider, for the first time, evidence on ALMP impacts in transition countries and, to a lesser extent, in developing countries. While very few evaluations existed outside the OECD at the 17 time of the 1999 report, 39 of the new studies come from developing and transition countries. A larger body of evaluations will be needed to draw more definitive conclusions about the impacts of ALMPs in non-industrialized countries but, nonetheless, we now have a large enough sample to make some initial observations. The sample of evaluations included in our review is summarized in Table 4. A critical question for our review concerns the quality of the evaluations. We have included only those evaluations that utilize a control group methodology. This filter applies both to the studies included from the 1999 report and to all new ones.7 While a control-group methodology is necessary for ensuring valid impact evaluation results, it is admittedly not a sufficient condition. We have also not included studies that raised obvious concerns about methodology and/or analytical rigor. However, there is no doubt that the studies included in our analysis still vary in quality. Ideally, a careful review should be conducted of the methodology used for each study and the results weighted according to the rigor of the approach. Unfortunately, we could not undertake this intensive quality control. As a result, the studies in this review are included on the basis of a fundamental, but limited, quality screen. The evidence is reviewed for seven ALMP categories: employment services, training for the unemployed, training for workers in mass layoffs, training for youth, wage and employment subsidies, public works, and micro-enterprise development/self-employment assistance. It should be noted that many interventions now include a menu of services so classifying programs can be problematic.8 Moreover, training programs cannot always be easily classified into one of the three categories we have used.9 For each type of intervention, we summarize the main conclusions drawn in the 1999 study; report on the findings of the new evaluations; assess the evidence pertaining to key program issues; and illustrate particular programs that may be of special interest, usually because of positive evaluation results. In terms of indicators, we focus on whether the post-program employment and 7Some evaluations reported in Dar and Tzannatos (1999) did use a control group methodology and these have been excluded from the present analysis. 8 In multiple-intervention programs, if evaluation results can be identified for separate types of interventions, each is included as a program in the review. 9 If the clear target is neither workers in mass layoffs or youth, programs have been placed in the general "training for the unemployed" category. 18 Table 4: Number of Evaluations by Type of Program, Country Group, and Period 1999 Study New Evaluations Total Developed Developing All Developed Developing All Developed Developing Total and and and Transition Transition Transition Employment services 15 2 17 4 5 9 19 7 26 Training for unemployed 16 3 19 19 11 30 35 14 49 Training for Mass layoffs 6 0 6 1 2 3 7 2 9 Training for youth 7 0 7 7 5 12 14 5 19 Wage and employment 6 1 7 11 5 16 17 6 23 subsidies Public works 6 3 9 2 9 11 8 12 20 Micro-enterprise development 5 2 7 4 2 6 9 4 13 and self-employment assistance Total 61 11 72 48 39 87 109 50 159 19 earnings of participants differ significantly from those characterizing the control group members.10 It should be noted that effects on both employment and earnings are not calculated in every evaluation. In particular, many studies consider employment impacts only. This is unfortunate in countries where unemployment benefits do not exist since impacts on earnings may be a more revealing indicator. The 87 new evaluation studies are listed in Annex 2. For each study, we include the design of the program, evaluation methodology, and the results of the impact evaluation.11 4.1. EMPLOYMENT SERVICES We examine evaluations for 26 programs that provide employment services for job- seekers. These programs comprise different types of services including job clubs (search techniques, support), counseling, testing and assessment, brokerage services, etc. All are designed to prepare job-seekers for employment and to improve the information on job opportunities. Of the 26 programs (17 from the 1999 study plus nine new ones), seven are from developing/transition countries. All new evaluations except one are quasi-experimental. 4.1.1 Assessment of Results The 1999 study was heavily based on the experience of industrialized countries. It drew favorable conclusions regarding employment services relative to other active labor market interventions. Generally, these services were seen to be relatively inexpensive yet demonstrated impacts that were no less effective than alternative programs. A similar assessment was drawn by the OECD (Martin and Grubb, 2001). According to Dar and Tzannatos (1999), effectiveness of employment services depended largely on whether the economy was growing or in a recession and on the availability of public funds. Programs that did not yield positive results were generally in areas with rising unemployment rates, while economic conditions have been usually favorable where programs have succeeded. The 1999 review also concluded that impacts varied with the target group; in particular, they did not 10In the case of micro-enterprise development/self-employment assistance, the impact indicators are survival rate of the business and earnings. 11Details on the programs and results from the first wave of evaluations can be found in Appendices 5.1-5.7 of Dar and Tzannatos (1999). 21 seem to improve the employment prospects or wages of youth significantly while some studies indicated a positive effect for women. Table 5 provides a summary of our latest findings on employment services, after adding the nine new evaluations to the 17 taken in the original review. In this table, and similar tables for the other interventions included in this review, we categorize the "bottom line" of the evaluations in terms of whether estimated impacts on employment and earnings were positive or not. As the table indicates, the overall impact of employment services on the subsequent labor market position of participants ­ relative to control group members ­ tends to be favorable. Of the 21 programs where we could determine an employment effect, 16 were positive and 11 of the 16 with clear earnings effects were positive, as well. When we look at non-industrialized countries, the limited evidence suggests that the overall positive results apply, at least to transition countries where most of the evidence exists. Six of the seven conclude that the employment effect is positive. The evidence on earnings impacts is less clear, with only four evaluations on which to base conclusions. As we will see below, however, the new evaluations in developing countries do raise some questions about the impacts of these programs, including who benefits. Table 5: Summary of Evaluation Results for Job Search Assistance/Employment Services No. of Evaluations Impact on Employment (*) Impact on Earnings (Number -- all studies) (Number -- all studies) 1999 New Total Positive Non- Not clear Positive Non- Not clear study positive positive (**) (**) Developed countries 15 4 19 10 4 1 9 3 1 Transition countries 2 3 5 5 - - 2 1 - Developing countries - 2 2 1 1 - - 1 - All studies 17 9 26 16 5 1 11 5 1 (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on employment and earnings impact do not add up to sample total because some studies did not consider both types of impact. (**) Not statistically significant, no effect, or negative results. 4.1.2 New Evaluations The nine new studies include four from industrialized countries, three from transition countries, and two from developing countries. An evaluation of Great Britain's "New Deal 22 for the Young Unemployed" shows important positive results. This program provides mandatory job search assistance and other services to 18-24 years olds with six months of unemployment (see Box 3 for details). The evaluations of the initial phase show that young unemployed men are about 20% more likely to find a job after the program (Van Reenen, 2003) and that the job search assistance component is responsible for some of this effect. The cost-benefit assessments are positive. Evaluations (one experimental, one quasi- experimental) from the United States show that employment service programs tended to have positive impacts on the speed of re-employment, on earnings, and on the reduction of unemployment benefits; however, impacts across states on post-program employment and earnings were uneven (Decker et. al., 2000; Jacobson and Petta, 2000). Box 3: Britain's New Deal for the Young Unemployed The New Deal for the Young Unemployed aims to help young people (between 18 and 24 years) who have been unemployed and claiming jobseeker's allowance for six months or more, to find work and to improve their longer-term employability. The program, introduced in 1998, combines initial job search assistance followed by various subsidized options including wage subsidies to employers, temporary government jobs, and full-time education and training. Participation in the program is mandatory and individuals who refuse to participate risk losing their entitlement to benefits. The path of a participant through the New Deal is composed of three main steps. On assignment to the program, the individual starts the first stage of the treatment called the "Gateway". This lasts for up to four months and is composed of intensive job-search assistance and short basic skills courses. Each individual is assigned a "personal advisor", a mentor who they meet at least once every two weeks to encourage and enforce job search. The second stage is composed of four possible options. There is the employer option ­a six-month spell on in subsidized employment. Second, an individual can enroll in a stipulated full-time education or training course and receive an equivalent amount to the job seeker's allowance for up to twelve months. Third, individuals can work in the voluntary sector for up to six months. Finally, they may take a job on the Environmental Task Force (essentially government jobs) and be paid a wage or allowance at least equal to the job seeker's allowance. Once this option period is over, individuals who have not managed to find a job or leave the job seeker's allowance pool for some other reason enter the "Follow through", which is third stage of the program. This is a process similar to the Gateway, taking up to 13 weeks, where job-search assistance is the main treatment provided. Recent evaluations of the program show that young unemployed men are about 20% more likely per period to gain jobs as a result of the New Deal. Part of this effect is due to subsidized jobs, and part (at least one-fifth) a pure "Gateway" effect. Evaluations also show that the job search assistance element of the New Deal is more cost effective than the other ALMP options as there is no subsidy involved. Unfortunately, results for women are not so clear because of sample size limitations. Source: Van Reenen (2003) and Blundell et al. (2001). 23 The new transition-country evaluations come from the Czech Republic, Macedonia, and Romania. In both the Czech Republic and Romania, employment services had positive impacts on reducing unemployment duration, including for groups that tend to have longer unemployment spells (Terrell and Storm, 1999; Benus and Rodriguez-Planas, 2002). However, new evaluations from two Latin American countries raise some questions about the impact of job search assistance in developing countries. A study from Brazil shows that the use of the public employment service does not increase the probability of finding formal work (although it does have a weak positive impact on moving into informal jobs). It highlights the fact that, where there are large informal sectors, workers often prefer other channels of job search. This evaluation illustrates the limited reach that public employment services may have in many developing countries (Woltermann, 2002). Other studies based on Uruguay's experience show that the impact of labor market information services on youth employment prospects is linked to their education, with better educated young workers showing the greatest gains from these programs (Fawcett, 2001). 4.1.3 Summary Recent evaluations confirm the previous fairly positive results, especially in developed and transition countries. Employment services costs are relatively low and program results are often positive. The recent evaluations confirm that these programs have more positive impacts under favorable economic conditions. Again, it should be understood that job search assistance and related employment services are unlikely to have much effect where labor demand is weak unless they are combined with other types of interventions. Earlier evaluations indicated that some types of workers benefited more than others. The more recent studies offer no new evidence that employment services favor women more than men. On the other hand, the evaluations in Uruguay suggest that these services tend to be more effective for the most educated workers. A major question in the area concerns the relative effectiveness of public vs. private delivery of employment services. Finally, more research is needed to assess the success of job search assistance in developing countries where informal sectors are large and hiring typically takes place through informal channels. This informality, plus the relative weakness of public institutions, may affect the performance of job search assistance programs in these countries. 24 4.2 Labor Market Training for the Unemployed We examined 49 evaluations of training programs primarily aimed at the unemployed, and often specifically targeted at the long-term unemployed.12 These programs are quite diverse but most have the objective of skills development through classroom and/or on-the-job training, which can include gaining work experience. Some courses are not vocational in nature but are aimed at building self-confidence, basic job readiness, and enhancing knowledge of the job market. The length of training courses for the unemployed included in our review varies considerably, from 6 weeks to 24 months. Of the 49 evaluations (19 from the 1999 study plus 30 new ones), 35 are from industrialized countries with 10 from transition countries and four from developing countries. Eleven of the 14 evaluations from developing/transition countries are new. The dominant methodological design for the training evaluations is quasi-experimental, with most applying matching techniques to analyze employment-related outcomes. 4.2.1 Assessment of Results The 1999 review concluded that training programs, especially for the long-term unemployed, often did not have positive impacts and, in most cases the costs were found to be high compared to the benefits of the program. Results seemed to be dependent on the business cycle, with programs performing better when they were instituted during periods of economic expansion. The evaluations reviewed at that time also showed that the interventions with the most positive returns were smaller-scale, on-the-job-training programs that directly met labor market needs and were well targeted, especially for women and other disadvantaged groups among the long-term unemployed. These conclusions were heavily based on industrialized-country experience and were very similar to those drawn by the OECD (Martin and Grubb, 2001). The OECD study noted that most programs that demonstrated gains for participants did so through improved employment opportunities rather than through higher hourly wages. Table 6 summarizes our latest findings on training for the unemployed, based on 19 studies drawn from the 1999 study and 30 new ones. On an aggregate basis, the table 12The definition of long-term unemployed differs by country. However, most use either a six- or twelve-month threshold. 25 suggests that these programs often do have a positive impact on the future employment of participants ­ this was the case in 28 of the 39 studies where an effect could be determined. The impact on future earnings was much less favorable with only 15 of 29 studies where a clear effect could be determined showing a positive result. As we note below, few studies included cost-benefit analyses. As Table 6 shows, the effects of training programs for the unemployed seem to be most positive for developed and transition countries. In the case of transition countries, all programs had positive employment impacts although in three of the five programs where earnings effects could be determined, they were neutral or negative. Of the four developing country evaluations, only one showed any gains in terms of employment or earnings. Table 6: Summary of Evaluation Results for Training Programs for the Unemployed No. of evaluations Impact on employment (*) Impact on earnings (Number -- all studies) (Number -- all studies) 1999 New Total Positive Non-positive Not clear Positive Non-positive Not clear study (**) (**) Developed countries 16 19 35 17 8 - 12 8 - Transition countries 3 7 10 10 - - 2 3 - Developing countries - 4 4 1 3 - 1 3 - All studies 19 30 49 28 11 - 15 14 - (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on employment and earnings impact do not add up to sample total because some studies did not consider both types of impact. (**) Not statistically significant, no effect, or negative results. 4.2.2 New Evaluations According to the new evaluation studies, labor market outcomes are significantly affected by the type of training. On-the-job training programs show favorable effects for many industrialized countries (Australia, Belgium, Great Britain, and Sweden), while programs only with classroom training tended not to have positive effects on employment and earnings in many cases (Australia, Belgium, Germany, and Switzerland). The effects on employment probabilities also differ in terms of the training content. For instance, in European countries, training programs directed to foreign workers to learn the native language show large and significant positive effects. In Switzerland, while the general training and computer courses actually had a significant negative effect, German language courses had a very large significantly positive impact (Prey, 2001). Further 26 evaluation evidence shows that specialized training courses and training structured to lead to formal qualifications have can have large positive impacts. A good example of this is the National Plan of Training and Reemployment in Spain that provides specific skills training in agriculture, industry, management, etc. This program has shown positive effects, reducing the probability of remaining unemployed (Arellano, 2002). Another point emerging from the recent evaluations is the significantly better performance of training programs with employer sponsorship and taking place in enterprises (Hui and Smith, 2002). Unfortunately, most studies do not include information about the costs of training programs or any cost-benefit analysis. When costs are known, they seem to be higher than the benefits or higher than other alternative ALMPs. The effects of training programs on employment for transition economies are always positive. This is consistent with earlier evaluations in transition countries that showed moderate but positive impacts (Fretwell et al., 1999). Evaluations in Bulgaria, Poland, and Slovakia all conclude that there is a significantly positive short-run effect from training on the probability of leaving unemployment for both men and women; however, when medium- and long-term effects are analyzed, only women shows positive effects (Poland and Slovakia). In Romania and Macedonia, evaluations concluded that impacts on both employment and earnings were positive. Training programs in transition countries also seem to be more cost-effective. Further research is needed to confirm this apparent success in transition economies and to understand possible explanations. As noted above, the results from the few evaluations conducted in developing countries are less promising. Although the characteristics of training programs differ widely, evaluations from Argentina and Colombia have similar findings -- statistically insignificant (neutral) results on employment probabilities and earnings. In Mexico, however, there are mixed results for the Job Training Program for Unemployed Workers (PROBECAT).13. Wodon and Minowa (1999) found that the program did not have large positive effects on 13 PROBECAT is a short-term training program targeted at increasing earnings and employment for unemployed and displaced workers. Trainees receive minimum wage during the training period (6 months) and the local employment office provides placement services. 27 employment probabilities and that it was not favorable from a cost-benefit perspective.14 A more recent evaluation finds more positive overall employment impact although this varies across personal characteristics, regions, and institutions (Calderon-Madrid and Trejo, 2001). According to these results, on-the-job training is more effective than classroom training, and, training centers run by the private sector outperform government-run ones. Evidence from these Latin American studies shows that positive impacts on employment and earnings of training programs are largely confined to women (Argentina, Colombia, Mexico). 4.2.3 Summary The new wave of evaluations reinforces the view that the record for training unemployed workers is mixed. There are many programs with positive results, especially in terms of increasing employment probabilities (as opposed to wage rates), but others appear to offer little benefit to participants. The evidence suggests that the design of the program is critical for ensuring favorable outcomes. A number of studies observe that on-the-job training and employer involvement and sponsorship are associated with more positive outcomes than classroom training and programs that do not have connections to the private sector. Studies also highlight the value of linking training with formal qualifications. Some earlier evaluations had found that unemployed women often received more benefit from training programs than men. The evidence from the recent evaluations on this point generally supports this but there are exceptions. Studies in developing and transition countries find that most of the positive impact is confined to women; on the other hand, the evaluation of the Spanish training program concluded that women, along with older workers and disabled workers, benefited the least. Finally, the recent evaluations we have studied confirm that programs aimed at the unemployed are more effective, at least in the short run, when the economy is performing well. The new evaluations have provided some insights into how well training programs can assist unemployed workers in transition and developing countries. However, they raise important questions as well. Why are the evaluations so consistently positive in transition 14This evaluation was a reevaluation of two previous studies in 1992 and 1994. The results of this third evaluation led to different findings and policy conclusions. 28 countries in terms of improving the employability of participants? On the other hand, it seems that the few programs in developing countries that have been evaluated have less impact than in either developed or transition countries. If this is indeed the case, do these programs need to be adjusted to reflect the large informal sectors and low administrative capacity in many of these countries? Many more careful evaluations will be needed to address these questions. Finally, more information about cost-effectiveness is needed in all countries. Unfortunately, most recent evaluations have had less emphasis on costs, compared to impacts. 4.3 Retraining Programs for Workers in Mass Layoffs Retraining programs specifically for workers involved in mass layoffs face special challenges. First, workers who lose their jobs in large-scale layoffs ­ from enterprise or industrial restructuring, privatization, etc. -- tend to be concentrated geographically, often in areas where local economies are in decline. They generally have a stronger history of employment than a typical pool of unemployed workers, but often their skills and experience are specific to a particular industry or occupation where labor demand is declining. So, in principle, retraining should be a promising option. Unfortunately, the body of evaluation evidence on retraining programs specifically for workers in mass layoffs is small. This is despite the fact that many countries, including most transition economies and a number of developing countries, have invested in retraining to support the structural reforms launched during the last decade. One reason for this small sample is that much of the training, especially in transition countries, to address large-scale restructuring has been carried out under general programs. Where there are evaluations, then, we have included them in the preceding sub-section. In terms of specific programs for retraining for workers in mass layoffs, we examine nine studies, including six from the 1999 report and only three new ones. Seven are from developed countries, with one each from a developing country and a transition country (both new). 4.3.1 Assessment of Results The 1999 review drew relatively unfavorable conclusions about the impact and cost- effectiveness of retraining programs specifically for workers in mass layoffs. Control-group 29 evaluations showed that some programs might result in modest increases in reemployment probabilities for participants, but this result was often statistically insignificant. There were very few examples of earnings gains and, often, evidence of losses from participating. Overall, Dar and Tzannatos (1999) concluded that these programs compared unfavorably in terms of impacts to training programs for the unemployed. According to their results, if retraining programs are to be used, they should be small- scale and targeted towards those subgroups who can benefit the most from them. Table 7 summarizes our updated results on retraining for workers in mass layoffs. It is based on the six studies from the 1999 review plus three new ones. Some programs do appear to have a positive employment impact but almost as many either have no significant effect or even a negative one. It should be noted that the small number of evaluations does limit the definitiveness of any conclusions that can be drawn. This also precludes any possibility of comparing results by development category. Table 7: Summary of Evaluation Results for Retraining Programs for Workers in Mass Lay- offs No. of Evaluations Impact on Employment (*) Impact on Earnings (Number -- all studies) (Number -- all studies) 1999 study New Total Positive Non-positive Not clear Positive Non-positive Not clear (**) (**) Developed countries 6 1 7 3 3 - - 1 3 Transition countries - 1 1 1 - - 1 - - Developing countries - 1 1 - - 1 - 1 - All studies 6 3 9 4 3 1 1 2 3 (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on employment and earnings impact do not add up to sample total because some studies did not consider both types of impact. (**) Not statistically significant, no effect, or negative results. 4.3.2 New Evaluations The three new evaluations come from Australia, China, and Bosnia and Herzegovina. They provide a mixed picture with a very positive evaluation in Bosnia and Herzegovina, a negative one in Australia, and mixed results in China. The Australian study evaluates the impact of retraining workers laid off from the textile, clothing and footwear sectors. These layoffs resulted from displacements due to trade 30 liberalization. Training, and specifically the length of the program, had significant negative effects on the likelihood of finding a job for both men and women (Weller, 1999). It should be noted that many of those laid-off had no formal qualifications and were not literate in English. While the evaluation did not formally address selectivity issues, the authors conjecture from their analysis that retrenched workers who gained the most from retraining were those with higher skill levels at retrenchment and who could build their skill repertoire to adapt to new contexts. The Chinese evaluation is very important, given the context in that country of large- scale displacement due to the massive restructuring of the state sector and other structural changes taking place. The study assesses the impact in two large cities of retraining programs for workers laid off from state-owned enterprises. In one of the cities, participation in retraining led to increases in employment while, in the other city, the employment effect was insignificant. In neither case was there a significant effect on earnings. The analysis did highlight three program design features that were associated with higher employment rates for the laid-off workers: longer-duration programs, some worker contribution to training costs, and programs offered outside the official Labor Bureaus. Bosnia and Herzegovina appears to have had a successful reemployment experience with the Emergency Demobilization and Reintegration Project (see Box 4). It should be noted that this project was carried out in a severe post-conflict context and caution should be taken in terms of its applicability to more normal situations. This project provided a range of instruments, including education and retraining as well as employment services, to assist in the reintegration of demobilized soldiers and displaced workers into the civilian workforce and to increase economic productivity.15 The results of the quasi-experimental evaluation show that the program increased the likelihood of employment by 43% and had a major impact on wages as well (Benus et al., 2001).16 Although the evaluation did not separate out the effects of specific instruments, retraining was the major intervention and presumably 15Requirements of the Dayton Peace Accords called for a balanced reduction of 425,000 men and women. Around 1,250,000 refugees were returning to the country, adding to the 900,000 people who were unemployed due to war damage to their place of employment. 16In a multivariate regression, monthly earnings were 98 DM higher for participants than the control group. The mean earnings for participants was 178 DM, so this program effect accounted for 55% of their observed earnings. 31 accounted for much of the impact. The program had a positive statistically significant impact on employment and earnings for all subgroups by gender, age, and the level of education. Unfortunately, the evaluation methodology does not include estimates of deadweight or substitution effects which may be significant given the design of the program. 4.3.3 Summary The small number of evaluations precludes any definitive assessment of this important type of intervention. Certainly, the addition of just three new evaluations cannot provide the basis for an informed reconsideration of the earlier, and largely unfavorable, evidence. The variation in the results of the new evaluations does underline that the extent to which retraining programs can assist in the reemployment of displaced workers is not predetermined. Design seems to matter. The Bosnia and Herzegovina case, which has yielded the most favorable results, included integrated employment and training services, and strong sponsorship and commitments by employers. But the post-conflict context does raise questions about exactly how generalizable the results are to the more standard case of industrial displacement. A larger body of evaluations will be needed to more precisely identify the key factors for getting positive outcomes in these retraining programs. More information about costs would help to further assess the viability of these programs. Finally, we reiterate that a lot of training for workers involved in mass layoffs actually takes place under general retraining programs. In the case of transition countries, where the evaluation evidence for general programs has been relatively positive (see previous sub-section), retraining in response to large-scale restructuring may have been more successful than the limited evidence in this sub-section suggests. 32 Box 4: Bosnia & Herzegovina Emergency Demobilization and Reintegration Project (EDRP) The EDRP began in 1996 and completed program operations in 1999. The goal of the project was to assist in the reintegration of demobilized soldiers and displaced workers into de civilian workforce and to increase economic productivity by financing the implementation of active labor market programs. The project design incorporated four components: · Labor Market Information Data Base: Reestablishment, upgrading, and broadening of municipal-level labor market information data base. · Education and Retraining Services: Financing sub-project contracts for demand-driven education and retraining services. Contracts were granted to enterprises that agreed to provide on-the-job-training and to hire 80% of the training participants. · Counseling and Job-Finding Services: Providing technical assistance to develop counseling and job-finding services as well as sub-project contracts for the delivery of these services. · Management Assistance: Providing technical assistance and goods to develop and maintain two small management units. The total funding of the project was $7.75 million. Throughout the three-year implementation period, project funds were allocated between the two political entities: the Federation of Bosnia and Herzegovina, with Muslim and Croat ethnic majorities, and the Republika Srpska), with a Serbian ethnic majority. Each of these political entities managed and implemented its own project resources independently. For example, each entity maintained its own small Project Implementation Unit (PIU) as well as its own extension agents who promoted and monitored the counseling and training contracts. Shortly after the two PIUs were established, public solicitations were published in local newspapers to identify local enterprises and educational institutions with the capacity and willingness to train and find jobs for program participants. A total of 538 contracts were signed, the majority (86.6%) with enterprises that agreed to provide on-the-job training. The remaining contracts were split approximately equally between training institutions (6.9%) and counseling service providers (6.5%). There were a total of 22,989 participants, similarly distributed with 80.3% on-the job-training, 5.2% in institutional training, and 14.5% in counseling. The evaluation of the program showed very large and statistically significant impact on the likelihood of wage and salary employment and on earnings. The services provided by the project were effective in reintegrating demobilized soldiers into the economy. Source: Impaq International (2001) 4.4 Training Programs for Youth Our review includes 19 evaluations of training programs for youth.17 This includes seven evaluations from the 1999 study and 12 new ones. Of these, 14 are from developed countries and five are from developing countries. All of the developing country evaluations are new. Studies on these programs are relatively evenly split between experimental and quasi-experimental design. These types of programs ­ and their evaluations -- receive a great deal of attention given the prominence of unemployment among young people, especially disadvantaged 17The standard definition of "youth" is 15-24 year-olds. However, many of the programs evaluated are targeted at young people under 20 years old. 33 youth, in many countries.18 In most countries, the youth unemployment rate is between two and four times the adult rate (and up to eight times in a few cases). In developing countries, this ratio tends to be higher than in industrialized ones (O'Higgins 2003). Many programs target less-educated young people, often school dropouts, and/or youth from disadvantaged families. The main objective is generally to improve the employability of participants, through increased skills for the labor market. Interventions typically include some combination of on-the-job training, vocational (classroom) training, job readiness training, and/or internships. In many -- especially industrialized -- countries, these programs are being delivered through public-private partnerships. 4.4.1 Assessment of Results Dar and Tzannatos (1999) concluded that youth training programs had the poorest track record, when compared with training programs for the long-term unemployed and those displaced through mass layoffs. OECD reviews have drawn a similar conclusion (Martin, 2000). In almost all cases included in the 1999 review, participants did no better than the control group, either in their post-training employment probability or their earnings. Moreover, the cost-benefit analysis of several of the youth training programs suggested that the social rates of return were typically negative, both in the short- and the long-run. The policy implication to be drawn from Dar and Tzannatos (1999) and Martin (2000) is that earlier interventions at the schooling stage are likely to be more effective than trying to remedy education failures through youth training. This is consistent with a recent review undertaken by Godfrey (2003) for the World Bank. This review concludes that youth employment policy should shift from "curative" to "preventative" interventions; i.e., shifting from dealing with the consequences to the causes. Table 8 summarizes the findings of evaluations on youth training programs. It includes the seven studies included in the 1999 review, plus the 12 new ones. The negative track record reported by the earlier World Bank and OECD reviews remains relevant for 18Unemployment, of course, is only one indicator of labor market outcomes. In developing and transition countries with large informal sectors and limited social safety nets, it is often not the best indicator. Employment in the informal sector or some measure of "underemployment" would be desirable but it is difficult to gather data on these measures. O'Higgins (2003) suggests that, if systematic data were available, it would reasonable to assume relatively high rates for youth. 34 developed countries. Only one program demonstrated a positive outcome for each of the employment and earnings indicators; all the rest had either neutral or negative impacts according to the evaluations. On the other hand, the picture seems quite different for youth programs in developing countries, at least in Latin America where the new evaluations come from; note that there were no evaluations from any developing countries available for the 1999 review. As the table indicates, all five studies of youth training programs in developing countries appear to have had positive employment impacts and two of the three computing earning effects demonstrated positive impacts on that indicator. We discuss these new developing-country evaluations, including qualifications on the results, below. Table 8: Summary of Evaluation Results for Training Programs for Youth No. of Evaluations Impact on Employment (*) Impact on Earnings (Number -- all studies) (Number -- all studies) 1999 New Total Positive Non-positive Not clear Positive Non-positive Not clear study (**) (**) Developed countries 7 7 14 1 10 1 1 8 - Transition countries - - - - - - - - - Developing countries - 5 5 5 - - 2 1 - All studies 7 12 19 6 10 1 3 9 0 (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on employment and earnings impact do not add up to sample total because some studies did not consider both types of impact. (**) Not statistically significant, no effect, or negative results. Note that we do not have evaluations of specific youth training programs in transition countries. However, there are pieces of research, summarized in Godfrey (2003), that provide some evidence on this issue. Evaluations of training programs by Fretwell et al. (1999) showed some positive impacts on youth for Hungary and Poland (on employment but not earnings), and the Czech Republic (earnings but not employment). However, Godfrey (2003) cites evidence that any positive employment impacts are likely outweighed by the costs of the programs.19 19 The increases in employment probabilities for youth were in the 6-10% range for Hungary and Poland (Fretwell et al., 1999). 35 4.4.2 New Evaluations Looking first at developed countries, evaluation results from Canada, France, and Sweden show disappointing results, similar to those reported in the earlier review. In these countries, youth training schemes appear to have had no effect on the likelihood of getting a stable job and did not increase employment duration if a job was found. The one positive evaluation from an industrialized country related to the Job Corps program in the United States. This program is targeted at 16-24 year-olds, many without high school completion. Average duration of participation is eight months and services include vocational training, academic instruction, and other social services. The evaluation study found that the program generated positive and statistically significant earnings impacts beginning in the third follow- up year, which persisted through the end of the four-year period after program participation. The average earnings gain from participation was estimated to be 12%. The program also showed positive results in terms of cost effectiveness (Gritz and Johnson, 2001).20 While the evaluation results are favorable for Job Corps, in many ways it is a special, high-cost intervention that includes a range of services that are outside the normal umbrella of ALMPs. As Table 8 indicates, the results for the new evaluations in developing countries (i.e., Latin America) are positive. This is particularly true for the "Jovenes" programs which have been evaluated in Argentina, Chile, Peru, and Uruguay. These programs, described in Box 5, share common features: they are targeted at disadvantaged youth, combining training and work experience with other services including psychological development, vocational assessment, etc. Implementation involves important roles for civil society and the private sector and flexible, competitive, and decentralized service delivery. Funding varies: for example, the Chilean program is financed by government while in Uruguay, it is funded from a tripartite levy-grant scheme (Godfrey 2003). In all four countries where evaluations exist, the program demonstrated positive and significant effects on employment and earnings. In some cases, these net gains are estimated to be substantial. However, without dismissing the positive results, there are questions about the Jovenes programs. Some critiques emphasize that there are no incentives for funding 20When savings from reduced criminal activity among the program participants was taken into account, the study estimated that society benefits by about $17,000 for each participant. . 36 agencies to focus on quality of training (Castro, 1999). Second, there is not much evidence relating to cost-benefit analysis of the programs. However, the costs of these programs can be high. The evaluation in Argentina estimated that at least nine years of higher earnings due to the program would be required to show a positive net present value for the groups with statistically significant results (Aedo and Nunez, 2001). Third, the evaluations measure the impact of the programs in the very short-run (six months to one year after receiving treatment); little or nothing is known about the impact over time. Finally, the overall coverage of these youth training programs ­ and thus their aggregate impact on the youth unemployment problem -- is limited by their design and relatively intense package of services.21 4.4.3 Summary The new evaluations reinforce earlier negative findings about the effectiveness of youth training programs, at least in industrialized countries. With the exception of the very unique U.S. Job Corps program, the evaluations in these countries indicate that youth training rarely improves the employment and earnings prospects for participants. The favorable assessments of the Jovenes programs in Latin America, however, raise the possibility that the disappointing track record of youth training in industrialized countries may not apply in developing countries. The uniformly positive estimates of program impacts on both employment and earnings in the four countries where evaluations were undertaken suggest that the Jovenes approach may be a promising one. As we have noted above, this conclusion should be tempered by questions about the longer-term impacts, the costs, and the potential coverage of these kinds of programs. 21 For instance, the Peruvian Projoven program has trained approximately 30,000 youths over five years -- a significant number but still a low one, considering the total number of unemployed youth. 37 Box 5: The "Jovenes" Programs for Youth Training in Latin America The "Joven" program was initiated in Chile in 1990, and with the positive results there, similar programs were implemented in Argentina, Colombia, Peru, and Uruguay. They are aimed at youth from low-income families, with low educational attainment, and little or no job experience. In some cases, preference is given to household heads with children. In general, the programs consist of training, work experience, and various services, including basic literacy and job search assistance. The training component is aimed at helping participants to attain a semi-skilled level in specific trades that are in demand in the labor market. The work experience component has to be carried out in a firm specially engaged for that purpose. The firm assumes a tutoring role under the supervision and responsibility of the executing unit, but is not obliged to remunerate trainees or to employ them after the program. There are variants oriented towards self-employment. The training and work experience normally lasts about six months -- 200 to 400 hours of training and two to three months of work experience. Participants get a maintenance and transportation subsidy. Although the implementation of the programs varies slightly across countries, there are some key common characteristics: · Coordination by the state, civil society, and the private sector. The State assumes responsibility for the design, supervision, technical control, and full or partial financing of programs. Implementation is by state agencies and civil society, which receives support services to develop their operational, technical, and administrative capacity. The private sector provides spaces for work experience and, in so doing, ensures that the training delivered meets real market needs. · Targeting. The design of the program, including geographical location, duration of the training package, qualification level achieved, conditions for work experience, amount of subsidies, etc., is intended to ensure that it reaches the target population and not "borderline" social groups. · Decentralized implementation, regulated by market mechanisms. Maximum flexibility and the decentralization of training services is achieved by using competitive bidding. In Chile, for example, the program involves about 1,000 training providers including private training centers and NGOs. Proposals are considered on both technical and financial criteria. · Integrated approach. Training goes beyond specific occupational skills and incorporates curriculum to improve communications, personal relations, and self-esteem. Evaluations of the "Jovenes" programs show that in all the countries, the program have been successful in terms of improving labor market outcomes for participants. Sources: International Labour Organization (2003) and Godfrey (2003). The received view that youth unemployment is most effectively and efficiently addressed through preventive policies (i.e., through improving education outcomes) rather than through training programs remains valid. However, the positive evaluations for the Jovenes and U.S. Job Corps programs do offer some lessons for what is required for successful youth interventions. These programs have in common an approach that involves a combination of services where training is provided in a comprehensive package that also includes basic education where needed, employment services, and relevant social services (including residential services in the case of Job Corps). Work or internship experiences also seem to be important. However, it is important to acknowledge that these comprehensive 38 service approaches involve tradeoffs: they are costly and cannot easily reach large numbers. Another successful design feature seems to be the use of private providers; all positive evaluations of youth training programs in developed and developing countries found that private trainers performed better than public institutions. 4.5 Wage/Employment Subsidies Wage and employment subsidies are designed to subsidize an employer's cost of hiring. These subsidies can take various forms including contributing part of the worker's salary or offsetting social security contributions for a specified amount of time. The level and duration of these subsidies can vary significantly. Subsidies are typically targeted at particular groups, such as the long-term unemployed, disabled workers, and young people. In a growing number of industrialized countries, subsidies are being offered directly to welfare recipients in order to encourage them to work. In addition to this labor supply effect, subsidy programs are seen as compensating employers for their screening, orientation, and initial training costs with the expectation that the employee's productivity will have increased enough by the time the subsidy period is over for the firm to continue the employment relationship. There also is a particularly strong social element to these programs even if there is no net employment gain (Martin 2000). Subsidies are a relatively straightforward way to get long-term unemployed or disadvantaged workers into the labor market, and governments sometimes use this instrument even if the job lasts only for a limited period of time (and perhaps at the expense of unsubsidized workers). We consider 23 evaluations of wage and employment subsidy programs. These include 7 from the 1999 review and 16 new ones. Of these, 17 are from industrialized countries, five are from transition countries, and one comes from a developing country. Five of the six evaluations in developing and transition countries are new. Five of the evaluations are based on experiments while the rest use a quasi-experimental design. 4.5.1 Assessment of Results The reviews by Dar and Tzannatos (1999) and Martin (2000) concluded that wage and employment subsidies are unlikely to be effective, with substantial deadweight and substitution risks. Most programs showed that participants were less likely to be employed and earned less than those in the control group after the subsidy ended. One exception was 39 the U.S. Job Training Partnership Act program where subsidies combined with on-the-job training offered significant benefits to single mothers and, to a lesser extent, males. These earlier reviews were based almost exclusively on evaluations in industrialized countries. A major risk of wage subsidies is that employers will use these programs even where hiring would have taken place anyway or to substitute subsidized workers for unsubsidized ones. If programs are to have real net positive impacts, it is important to target carefully and to monitor employers to reduce these substitution and deadweight effects. However, Martin (2000) points out that this involves a tradeoff in that intense monitoring and tight targeting conditions may make the program unattractive for employers and thus reduce the take-up rate. Table 9 summarizes the evaluations for wage and employment subsidies, based on the 23 available studies. The overall picture remains unfavorable: 14 of the 21 evaluations with results on employment impacts conclude that the effect was either neutral or negative and only five of 11 evaluations find that the subsidies had a positive impact on earnings. The table indicates that the positive findings almost completely come from industrialized countries where some program evaluations do show net employment and/or earnings gain. In fact, the new wave of evaluations in industrialized countries is more positive than the evidence considered by Dar and Tzannatos (1999).22 The evaluations from transition countries lead to uniformly negative assessments. The one evaluation from a developing country reports positive employment impacts. In any event, most evaluations do not fully estimate deadweight and substitution effects that can reduce overall program impact. One promising direction concerns programs in the U.S. and Canada where subsidies have been provided directly to welfare recipients as a supplement to their earnings. These are described below. 22The 1999 study evaluations for developed countries had two positive cases of employment impacts and 11 negative or neutral. Six of the 10 added in this study had positive results in terms of employment effects. 40 Table 9: Summary of Evaluation Results for Wage and Employment Subsidies No. of Evaluations Impact on Employment (*) Impact on Earnings (Number ­ all studies) (Number ­ all studies) 1999 New Total Positive Non-positive Not clear Positive Non-positive Not clear study (**) (**) Developed countries 6 11 17 6 9 2 5 4 1 Transition countries 1 4 5 - 5 - - 1 - Developing countries - 1 1 1 - - 1 - All studies 7 16 23 7 14 2 5 6 1 (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on employment and earnings impact do not add up to sample total because some studies did not consider both types of impact. (**) Not statistically significant, no effect, or negative results. 4.5.2 New Evaluations The new evaluations in industrialized countries show both negative (Belgium, Sweden) and positive results (Australia, Germany, Great Britain, and the U.S.). However, some of the programs with positive results have other elements besides the wage subsidy and the evaluations do not isolate the specific effects of the subsidies. For example, the Public Employment Program in Germany also includes training and the New Deal Program in Great Britain also has job search assistance services. Where employment impacts are positive, the magnitude of the effect is often modest. As noted above, programs that provide the subsidies directly to individuals as a supplement to their earnings seem promising. The objective of these programs is to use subsidies as an incentive to encourage work ­ as opposed to welfare payments -- as a way to achieve greater economic self-sufficiency. Canada and the U.S. have implemented and evaluated these type of programs. On average, the Canadian Self-Sufficiency Project (see Box 6) increased earnings by more than 20% over the control group. Because the rules of the program prohibited people from simultaneously receiving the earnings supplement and income assistance, the program reduced income assistance payments by about $3,500 per family in the treatment group (Michalopoulos et. al. 2002).23. In the United States, for most groups, participants of welfare-to-work programs (see Box 7) had higher earnings and lower 23 However, other evaluation analysis using a general equilibrium approach shows that the impact of the project is not as good as it seems to be when evaluating only employment and earning effects on individuals (Lise, Seitz and Smith, 2003). 41 welfare payments than people in the control groups, but generally had the same combined income from earnings, welfare, and Food Stamps. The program also reduced annual welfare payments by similar amounts for all groups, with the effects ranging between $200 and $600 (Michalopoulos, et. al. 2001). The available evaluation evidence for wage subsidy programs in transition countries is negative. The evaluation of wage subsidies in Poland is particularly unfavorable for males. In Slovakia, neither men nor women benefited from the subsidies. The developing country experience is limited to the Argentina Proempleo experiment. In this evaluation, participants (low-income and registered in workfare programs) were divided into three groups ­ receiving vouchers only, vouchers plus training, and the control group. The impact of the voucher was not evident until 18 months after the program; at this point, 14% of participants reported private-sector wage employment compared to 9% for the control group (Galasso, Ravallion, and Salvia, 2001). There was no positive effect, however, on earnings. Take-up by employers was low, apparently because of associated costs (social charges, eventual severance payments, etc.) for the enterprise. 4.5.3 Summary The clear majority of subsidy programs do not appear to have net positive impacts on the longer-term employability or earnings of participants. This is particularly the case for developing and transition countries where the limited evaluation evidence is uniformly negative. But there are exceptions. The picture in industrialized countries is more favorable, especially on the basis of the recent evaluations. There is some new evidence of effective programs, when employers use the subsidy to screen future workers or when provided directly to individuals through an incentive system to work, at the risk of losing welfare benefits. However, all evaluations of this type of labor market intervention are beset by the challenges of factoring in deadweight and substitution effects that are especially relevant in the case of subsidies. Most recent evaluations have not incorporated these costs. Finally, programs may be most effective when they combine the subsidy with other components such as training or job search assistance. 42 4.6 Public Works Public works programs are increasingly known by other names including temporary community projects, labor-intensive projects, and workfare to reflect the fact that are no longer exclusively government-delivered infrastructure projects. These projects often are delivered by NGOs or the private sector and they may involve diverse activities including services for the community. All of these projects have multiple objectives including transferring income to poor families especially during difficult times, providing infrastructure or services, and creating jobs. Most programs offer participants low-wage, short-term jobs in construction, rural development, community services, etc. The length of these jobs can vary but typically is between three months to one year. These programs are widely used by countries at all stages of development. In many low-income countries, they are the most heavily used labor market intervention. Much of the policy debate over the effectiveness of public works programs has focused on the safety net, or anti-poverty, objective. The central question for this review, however, concerns the employment impacts of public works and, specifically, whether participation improves future employment and earnings prospects. We consider 20 evaluations, nine from the 1999 study and 11 new ones. While the first wave of evaluations were primarily from industrialized countries (six of the nine), nine of the 11 new ones come from developing and transition countries. All but one of the evaluations is based on quasi- experimental design. 4.6.1 Assessment of Results The 1999 review concluded that public works primarily provide short-run benefits as a temporary safety net but that they are generally not effective as an active labor market program that improves longer-term employment prospects.24 There can be temporary declines in unemployment as a result of individuals joining the program but the body of evidence available to Dar and Tzannatos (1999) suggested that they do not increase future labor force attachment or employment probabilities. These conclusions were supported by the transition-country evaluations carried by Fretwell et al. (1999). 24For a review of the evidence on the effectiveness of public works as a safety net, see Subbarao (2003). 43 Table 10 summarizes the evidence from the 1999 review and the new evaluations on impacts of public works programs. The unfavorable earlier assessment of public works as an employment program still applies although the new evaluations are more positive than the ones considered in 1999. Overall, seven of the 18 studies with estimates of the employment effect have positive results. Five of these come from the new studies. Most evaluations do not analyze the earnings effects ­ of the four that do, none are positive. As the table indicates, the evaluation evidence is negative for the majority of programs in industrialized countries and mixed for the transition country programs. The only evaluation available for a developing country (Argentina) is inconclusive in terms of post-program labor market outcomes. Table 10: Summary of Evaluation Results for Public Works Programs No. of Evaluations Impact on Employment (*) Impact on Earnings (Number -- all studies) (Number -- all studies) Developed countries 1999 New Total Positive Non- Not clear Positive Non- Not clear study positive positive (**) (**) Transition countries 6 2 8 2 7 - - 1 - Developing countries 3 8 11 5 4 1 - 3 - All studies - 1 1 - - 1 - - 1 Developed countries 9 11 20 7 11 2 0 4 1 (**) Not statistically significant, no effect, or negative results. (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on employment and earnings impact do not add up to sample total because some studies did not consider both types of impact. 44 Box 6: The Self-Sufficiency Project in Canada The Self-Sufficiency Project (SSP) is a research and demonstration project designed to test a policy innovation that makes work pay better than welfare. The program offered a temporary earnings supplement to selected long-term income assistance recipients in British Columbia and New Brunswick. The earnings supplement was a monthly cash payment available to single parents who had been on income assistance for at least one year and who left income assistance for full-time work. The main key features of the SSP earnings supplement are: · Substantial financial incentive. The supplement equaled half the difference between a participant's earnings and an "earnings benchmark". During the first year of operations, the benchmark was between $30,000 and $37,000. Unearned income (such as child support), earnings of other family members, and number of children did not affect the amount of the supplement. The supplement roughly doubled the earnings of many low-wage workers (before taxes and work-related expenses). · One year to take advantage of the offer. A person could sign up for the supplement if she found full- time work within the year after random assignment. If she did not sign up during that year, she could never receive the supplement. · Voluntary alternative to welfare. No one was required to participate in the supplement program. After beginning to receive the supplement, participants could decide at any time to return to income assistance, as long as they gave up supplement receipt and met the income assistance eligibility requirements. Because the evaluation of SSP assigned people to the program and control groups at random, the impact of the supplement offer is measured as the difference in employment, earnings, income, and other outcomes between the two groups. One- third of the long-term welfare recipients who were offered the SSP earnings supplement worked full time and took up the supplement offer. Furthermore, by the end of the first year after random assignment, treatment group members were twice as likely as control group members to be working full time and the effect of SSP on employment continued to be strong through most of the follow- up period. As a result, SSP increased the average person's earnings by nearly $3,400, or more than 20% over the earnings of the average control group number. Because the rules of SSP prohibited people from simultaneously receiving the earnings supplement and income assistance, the program reduced income assistance payments by about $3,500 per family in the program group. When people left income assistance to receive the earnings supplement, they replaced their income assistance payments with SSP supplement payments. Over the entire follow-up period, treatment group members had on average about $6,300 more in combined income from earnings, income assistance payments, and earnings supplements than control group members. Finally, the effects of SSP on employment, welfare use, and income were small after parents were no longer eligible for the supplement. Members of the program group could receive supplement payments for up to three years, and the program's effects were strong throughout the period when parents were eligible for the supplement. Although the program's effects were smaller at the end of the follow-up period, this finding does not change the fact the program group members gained considerable work experience because of SSP and their families benefited from the increased income they gained while the supplement was being paid. Despite these positive results, general equilibrium approaches to the effects of the program, in contrast with partial equilibrium, show an important deadweight loss of the program, with the consequence that the benefits of SSP were considerably lower. In New Brunswick, for example, the benefits are approximately one-tenth the size of the net gain predicted by the partial equilibrium analysis. Sources: Michalopoulos et. al. (2002) and Lise, Seitz and Smith (2003). 45 Box 7: Evaluation of Welfare-to-Work Programs in the United States In 1996, the U.S. Congress passed the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), replacing the 60-year-old Aid to Families with Dependent Children entitlement program (AFDC) with Temporary Assistance for Needy Families (TANF). Under this new mechanism, states received block grants and flexibility to design their welfare programs. A number of states replaced voluntary welfare-to-work programs that emphasized education and training with mandatory programs that stressed quick employment. PRWORA also required a growing percentage of the welfare caseload to be working or participating in work- related activities and it imposed a five-year time limit on how long most families could receive federal support. In 2001, the U.S. Department of Health and Human Services implemented an evaluation of 20 welfare-to-work programs in eight states (Georgia, Ohio, Michigan, Oklahoma, Florida, Oregon, California and Minnesota). The evaluated programs all required some portion of the welfare caseload to participate in a welfare-to-work program or risk losing some or all of their welfare benefits through sanctions. While all the programs began operating prior to the passage of PRWORA, the earliest began in 1985 and the latest are still in operation. The programs vary in their approach to helping welfare recipients find work; five programs encouraged or required nearly all individuals to look for work, seven focused on basic education for most participants, and eight used a mix of the two approaches, encouraging or requiring more job-ready participants to look for work but allowing others to build skills through basic education. Manpower Demonstration Research Corporation (MDRC) conducted the evaluation using experimental research design in which individuals were assigned at random either to a program group or to a control group. According to these findings, people in most program groups had higher earnings and lower welfare payments than people in the control groups, but generally had the same combined income from earnings, AFDC, and Food Stamps. In addition, the programs increased earnings about as much for the more disadvantaged groups - long-term recipients, high school non-graduates, families with three children or more, and people with no recent work experience - as for the less disadvantaged groups. Nevertheless, the more disadvantaged groups earned much less than others. Finally, the study found that employment-focused programs tended to be more effective than education-focused programs for the more disadvantaged groups. While programs with an education focus rarely had large effects for these groups, programs that provided a mix of activities tended to help the broadest range of people. No information about deadweight loss effects is provided in this study. Source: MDRC (2002). 4.6.2 New Evaluations Recent evidence from industrialized countries includes only two evaluations, from Belgium and from Canada. The Belgian evaluation has unclear results. The assessment of the Canadian direct job creation programs involved various programs from 1977 to 1996. The evaluation found that participation enhanced employment and earnings for poor groups and women. The cost-effectiveness of these Canadian programs improved over time through better design, eliminating ineffective elements and expanding effective ones. 46 The eight new evaluations from transition countries present mixed results. In Poland, two evaluations concluded that public works had significant negative effects on exiting unemployment and on future employment; these impacts were strongest for men. One of these evaluations speculated that a consequence of these programs was to stigmatize participants as low-productivity workers. Evaluations in Slovakia, Slovenia, Ukraine, and Macedonia found that participation in public works had an immediate positive impact on post-program transition into employment. However, according to the studies in Slovenia and Ukraine, this positive effect only applied if participants found a job shortly after the program ended. In Slovenia, for example, there was no longer a positive program effect if no job had been found in three months. In Macedonia, the positive employment result was achieved at a very high unit cost. In Romania, participation in public works had no impact on either wage employment or self-employment. Although direct job creation programs through public works are the favored government response to unemployment in many developing countries, their labor market impacts have rarely been evaluated. We have only one study ­ a randomized experiment to assess the Trabajar program in Argentina (see Box 8). This program (second phase) was introduced in 1997 in response to the macroeconomic crisis and provided short-term work at relatively low wages, largely to workers from poor families.25 Ravallion et al. (2001) estimate impacts by comparing incomes of those who left the contracting program (usually involuntarily) against continuing participants and a control group of non-participants. They find that workers who left the program had large initial income losses relative to stayers or non-participants.26 However, it is not possible from this evaluation to answer the question at the center of our enquiry ­ i.e. whether participants improved their employment probability and earnings, compared to not having participated.27 25The evaluation of the safety net impacts of Trabajar was favorable. It was well-targeted on the poor and income gains for participants were equal to about one-half of the gross wage (Jalan and Ravallion 2002). 26There was evidence that ex-participants began to reverse these initial (within six months of leaving) losses in the second survey 12 months later. However, there were no subsequent follow-ups to observe longer-term impacts. 27Ravallion et al. (2001) do report qualitative data indicating that about one-half of participants felt that participation would expand their job opportunities. 47 Box 8: Argentina's "Trabajar" Program During 1995-1996, within the context of making longer-term macroeconomic labor market reforms, the government of Argentina began a series of short-term public employment programs to alleviate its growing poverty problem. Trabajar was started in March 1996, replacing a previous public employment scheme which was unsuccessful. A second phase was introduced in 1997. Trabajar provides low-wage work, targeted to the unemployed, on various small-scale community-level projects. The overall aim is to reduce poverty by providing paid work for unemployed workers from poor households and buy producing things of value to those households and other poor families in the local community. The main targeting mechanism is the low wage rate. The selection of an appropriate wage rate is critical to the success of reaching the target population. In the beginning, the direct benefit was a wage transfer of $200/month to every full-time worker in the program. Then this wage was $160/month, approximately 2/3 the average wage of the poorest 10% in the country. Low wages were used to make the program be attractive only to those people with low family incomes and few prospects for employment. Along with worker self-selection, the project selection process places an emphasis on geographically targeting poor areas to receive projects. Therefore, neighborhoods and municipalities of the target group are promoted as potential recipients for projects that in turn provide opportunities for work. The "Trabajar" program, supported by the World Bank, is managed by the Ministry of Labor and Social Security (MTSS) staff at the national, regional and provincial levels. The MTSS provides the menu of acceptable projects, as well as criteria and other instructions for project preparation, evaluation, approval, and supervision. MTSS assigns resources for each province using a formula which weighs most heavily the distribution of the unemployed poor with consideration for past implementation performance. Projects can be proposed by municipalities (66% of all the projects), NGOS (15%), national agencies, or private organizations. These projects include minor construction, repair, expansion, or remodeling of schools, health facilities, basic sanitation facilities, small roads and bridges, small dams and canals, community kitchens and centers, tourist centers, and low-cost housing. Projects must be completed within 4-6 months and can employ no more than 100 persons. The average project has had a cost of $70,000, employed 25 workers, and lasted five months. Sources: Eisenstadt (1998) and World Bank Institute (2000). 4.6.3 Summary Public works programs provide mainly short-term benefits in the form of a safety net and, when well targeted, they can be useful tools to fight against poverty by offering temporary employment. Establishing a low wage level is important to assure that the program reaches poor families and workers who really do not have alternative job options. As active labor market programs, public works are less effective. The majority of evaluations in developed and transition countries show that participants are no better off, and may be worse off, in terms of finding employment after the program. In some countries, negative post- program impacts may result from a stigma associated with past involvement in public works. However, there are some exceptions to this largely unfavorable track record. Also, it is important to monitor impacts well after program participation has ended. In some 48 evaluations, a positive employment effect disappears if the worker has not found a job immediately after the program. 4.7 Micro-Enterprise Development Assistance/Self-Employment Assistance These programs are intended to support the start-up and development of self- employment endeavors or micro-enterprises. They usually include some form of financial assistance (credits, allowances, grants), often supported by technical services such as training, counseling, and assistance in developing and implementing a business plan. Programs may screen participants on the basis of a feasible business plan. These programs may be targeted to particular groups or made more generally available to unemployed workers. There is only a small pool of evaluations which reflects the limited use of these programs compared to other ALMPs. We have 13 evaluations of micro-enterprise or self- employment assistance programs, seven from the 1999 report and six new ones. All but two use a quasi-experimental methodology. 4.7.1 Assessment of Results The earlier reviews by the World Bank and the OECD were somewhat tentative, given the limited number of available scientific evaluations. They did suggest that these programs can help a small subset of the unemployed population ­ specifically better educated, and often male. While these reviews provided some optimism about the potential effectiveness of self-employment/micro-enterprise assistance programs, there are a number of qualifications. Take-up rates are very low (5% or less, at least in OECD countries) because the large majority of workers do not see self-employment as a possibility. The high rates of new-business failure ­ whether assistance is provided or not ­ mean that these types of interventions entail considerable risk.28 Also, while deadweight and displacement costs may be important, these effects have generally not been fully considered. Table 11 summarizes the updated picture, adding the six new studies to the seven taken from the 1999 review. The orientation of many of these evaluations is to assess whether these programs can effectively support business start-up, rather than to evaluate 28Dar and Tzannatos (1999) had no data on whether assistance programs affected the survival rates of new businesses, relative to a control group. 49 whether they can help unemployed workers find employment and improve earnings. The table includes a business survival-rate indicator instead of the employment indicator used for the other programs.29 The evaluations show that these programs usually are associated with high rates of business survival but this is not a net indicator since it does not involve a comparison with a control group . Only five evaluations have results on earnings impacts (four from industrialized countries) with mixed results. Table 11: Summary of Evaluation Results for Micro-Enterprise Development/Self- Employment Assistance Programs No. of Evaluations Survival Rates of Impact on Earnings Micro Enterprises (Number -- all studies) 1999 study New Total High (*) Low (**) Positive Non-positive Developed countries 5 4 9 7 2 2 2 Transition countries 2 2 4 1 - 1 - Developing countries - - - - - - All studies 7 6 13 8 2 3 2 (*) Predominant effect either in the short or long term, depending on the study methodology. Totals on earnings impact do not add up to sample total because some studies did not consider both types of impact. (**) Not statistically significant, no effect, or negative results. 4.7.2 New Evaluations The six new evaluations include four from industrialized countries and two from transition countries. The results suggest that self-employment assistance programs can have favorable impacts on the limited group that takes them up. The Canadian program showed a statistically significant and sizeable positive earnings gain for participants in the year following the program. A recent Swedish study compares outcomes for the self-employment assistance program with an employment subsidy intervention and finds that the self- employment assistance program participants had a significantly lower probability of future unemployment.30 On the other hand, the German evaluation ­ while not examining traditional ALMP-evaluation impacts ­ had unfavorable results. Subsidized companies had a significantly lower survival rate than companies with otherwise similar characteristics. 29The implication of the employment-impact indicator is different in the case of this ALMP since individuals will report (self) employment as long as the business has not failed. 30Note, however, that the comparison in this study is not with a traditional control group. 50 Furthermore, those companies that did survive had no higher growth in the number of employees than the unsubsidized ones. The evaluations from transition economies come from Bulgaria and Romania and each has positive results. The Bulgarian self-employment assistance program increased the probability of re-employment by 43% and it appears that the program was successful in supporting sustainable businesses (Walsh, 2001).31 Participants in the Romanian program had net earnings and employment gains. 4.7.3 Summary The evaluation literature on the labor market impacts of ALMPs is thinnest in the case of micro-enterprise development and self-employment assistance programs. There are relatively few studies and of those that do exist, many are concerned with the program's effect on business development rather than on the future employment and earnings of participants. Furthermore, as noted above, the evaluations tend not to cover displacement and substitution effects which presumably could be significant in this type of intervention. With these caveats, it does appear that these programs can provide effective support for the small minority of unemployed workers who are interested in starting their own business. However, this finding is not universal ­ some evaluations show negative or insignificant effects. The literature does suggest that programs offering mentoring and business counseling, in addition to financial aid, are more likely to succeed than those only offering the latter. In the final analysis, much more evaluation needs to be undertaken to understand the impacts of programs to assist unemployed workers in starting their own business. This is particularly true in the case of transition and developing countries where little rigorous evaluation has been undertaken. 5. CONCLUSIONS The updated evaluation evidence on active labor market programs largely reinforces the conclusions drawn from earlier reviews. Some ALMPs do have positive impacts, with favorable cost-benefit ratios. However, in many cases, programs have not improved the 31However, the period between participating in the self-employment program and the evaluation is short. The author suggested re-evaluating the program one year later. 51 future employment prospects of participants and, when they have, they have not always done so in a cost-effective manner. Table 12 summarizes the findings from our review of over 150 evaluations. Employment services are generally the most cost-effective intervention: employment and earnings impacts are usually positive and, compared to other ALMPs, these employment services are inexpensive. Training programs for the unemployed can also have positive impacts on employment, though usually not on earnings. These programs are most effective when they are workplace-based. Other types of training programs ­ for workers involved in mass layoffs and for youth ­ address more serious challenges and generally have less favorable results. The interventions that are successful often feature an integrated package of services (education, employment, social, as needed) to complement the training. The evaluation evidence is also critical regarding the effects of direct job creation programs ­ wage and employment subsidies and public works. Recently, there have been some positive evaluations of subsidy programs but the overall picture is still unfavorable in terms of their net impact on the future employment prospects of participants. Public works can be a useful short-term safety net but, in most cases, they also do not lead to higher employment probabilities or earnings after the program has ended. Micro-enterprise development and self- employment assistance programs can be a useful tool for the small sub-set of workers who take them up; these are generally the better-educated among the pool of unemployed or eligible participants. The new dimension of our review has been to consider the impacts of ALMPs outside the industrialized countries, which have been the near-exclusive focus of earlier assessments. We have been able to partially address this issue with a growing body of evaluations in transition and (to a lesser extent) developing countries. While we can now offer some preliminary assessments, many more evaluations will be required to draw more definitive conclusions. 52 Table 12: Summary of Impact Evaluations Intervention Summary of Overall impact Comments Employment services Generally positive impacts on Programs have most positive impacts when employment and earnings in economy good. Impact limited where lack of developed and transition countries. labor demand. Costs relatively low so cost-benefit In developing countries, questions about ratio usually positive. coverage and effectiveness of these services, Very little evidence for developing given informal LMs. countries Training for unemployed Positive impacts on employment but Program effectiveness benefits from on-the-job no overall effect on earnings in training and employer involvement. developed countries. Women often seem to benefit more than men. Positive on both in transition Programs have most success when economy countries. good. Evaluations in developing countries not positive (but few studies). More information on costs needed. Retraining for workers in Often no positive impact on Better results may be achieved with integrated mass layoffs employment and earnings but there training and employment services. are exceptions. Very little evidence for transition and developing countries. Training for youth Very negative impacts on Youth employment problems more effectively employment and earnings in addressed through earlier, education-related developed countries. interventions. Positive impacts in developing Training most effective when combined with countries (Latin America), though other employment, basic education, and social few studies. services. No evidence from transition Successful programs require intensive services countries. and, thus, are costly. Wage/employment Overall negative impacts on Recent evaluations in developed countries more subsidies employment and earnings in favorable (e.g., welfare-to-work programs). developed and transition countries. Programs may be more effective when Very little evidence in developing combined with training. countries. Deadweight and substitution effects likely important but often not fully assessed. Public works Overall negative impacts on Can be effective as a short-term safety net for employment and earnings in the poor but generally not as program to developed and transition countries. improve future LM prospects. Very little evidence on LM impacts in developing countries, despite widespread use. Micro-enterprise /self- Not enough LM-oriented evaluations Very low take-up. employment assistance to determine overall employment and Some evidence of positive impacts for older, earnings impacts. better-educated individuals. Results likely better when technical and advisory services accompany financial aid. What can we say about the impacts of ALMPs in developing and transition countries? Many findings from industrialized countries do seem to apply broadly to transition countries 53 but ­ on the basis of what is still a small sample of studies -- this is not always true in the case of developing countries. The much larger informal labor markets and weaker capacity to implement programs may limit what some programs can achieve in terms of creating formal employment or increasing wages. The few evaluations in these countries for employment services and training programs for the unemployed are less positive than the (much larger) body of evidence in the OECD and transition countries. On the other hand, some youth training programs in developing countries have much more positive impacts than are seen in industrialized countries. It may be that such programs in these low-income labor markets have more potential because abundant supplies of skilled workers are not available. The sample of evaluations outside the OECD is still limited, especially in developing countries, and further studies will be needed to confirm these initial observations. While further evidence may reveal differences in the impact of ALMPs in countries at different stages of development, the ingredients for successful interventions probably do not differ. Comprehensive packages of services, programs that are oriented to labor demand and linked to real workplaces, and careful targeting are good examples. Finally, the evaluations underline the fact that program outcomes are usually better when the economy is growing. Knowledge on the impacts of ALMPs is improving but many policy-relevant questions remain, especially in the context of developing and transition countries. Four key issues for future attention can be highlighted, although the fact that these are gaps in our knowledge reflects practical difficulties in incorporating these issues into evaluation studies. First, what are the longer-term effects of these programs? Evaluations usually do not track post-program outcomes for more than one or two years so we do not know whether benefits dissipate or emerge after the evaluation period. Second, what are the general equilibrium effects? Since many studies do not fully estimate the deadweight, substitution, and displacement effects, we often do not know what impacts the programs may have on non- participants and on the functioning of the labor market. Third, are programs cost-effective? Impacts are only one-half of the story for evidence-based policy-making -- the ultimate economic criterion for resource allocation should be whether social returns exceed costs. However, many evaluations cannot offer definitive evidence on this because they do not fully consider program costs. Finally, the evaluation literature provides insights into what works 54 but far less on why. Ultimately, policy-makers need to understand what circumstances and design features explain effective outcomes for specific groups (Martin, 2000). Despite the mixed evaluation picture, governments, faced with the economic and social problems associated with large numbers of unemployed and poor workers, have little choice but to use active programming as one instrument in their response. They should be realistic about what ALMPs can achieve and allocate resources on the basis of cost- effectiveness. The challenge, then, is to learn from existing experiences, investing in programs that have positive returns and altering or dropping programs that do not. If ALMPs are going to be an economically useful policy, it is very important that governments carefully evaluate their own programs and introduce interventions on the basis of what works domestically and in other countries. 55 References Abadie, Alberto, Joshua Angrist and Guido W. Imbens. 2001. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Department of Economics Working Paper no. 99/16, Massachusetts Institute of Technology. http://www.econometricsociety.org/es/thejournal.html Aedo, Cristian and Sergio Nuñez. 2001. "The Impact of Training Policies in Latin America and the Caribbean: The Case of Programa Joven." Research Network Paper. Inter- American Development Bank. Washington D.C. Arellano, Alfonso. 2003. "Do Training Programmes get the unemployed back to work? A look at the Spanish experience". Universidad Carlos III de Madrid. http://www.iza.org/en/papers/summerschool/6_arellano.pdf Bell, Brian, Richard Blundell and John Van Reenen. 1999. "Getting the Unemployed Back to Work: The Role of Targeted Wage Subsidies," International Tax and Public Finance 6, no. 3: 339-360. Benus, J. and L. Orr. 2000. "Study of Alternative Quantitative Evaluation Methodologies". Working Paper. ABT Associates, Washington D.C. Benus, Jacob, James Rude and Satyendra Patrabansh. 2001. "Impact of the Emergency Demobilization and Reintegration Project in Bosnia & Herzegovina." Impaq International. Benus, J. and Rodriguez-Planas. 2002. "Impact of Active Labor Market Programs." Prepared for Government of Romania, Ministry of Labor and Social Protection. Betcherman, Gordon, Amit Dar, Amy N. Luinstra and Makoto Ogawa. 2000. "Active Labor Market Programs: Policy Issues for East Asia," Social Protection Discussion Paper no. 0005, January. Washington, DC: The World Bank. Betcherman, Gordon and Rizwanul Islam (Eds.). 2001. East Asian Labor Markets and the Economic Crisis. Washington, DC: The World Bank and the International Labour Office. Bidani, Benu, Chorching Goh and Christopher J. O'Leary. 2002. "Has Training Helped Employ Xiagang in China? A Tale from Two Cities.," March. Washington, DC: The World Bank. Bierhanzl, Edward. 1999. "Active Labor Market Policies and Measured Unemployment in Europe," Journal of Private Enterprise 15, no. 1: 109-23. 56 Bloom, Dan, Susan Scrivener, Charles Michalopoulos, Pamela Morris, Richard Hendra, Diana Adams-Ciardullo, Johanna Walter and Wanda Vargas. 2002. "Jobs First: Final Report on Connecticut's Welfare Reform Initiative," Mimeo February. New York: Manpower Demonstration Research Corporation. http://www.mdrc.org/publications/90/overview.html Bloom, Howard, Saul Schwartz, Susanna Lui-Gurr and Suk-Won Lee. 1999. "Testing a Re- employment Incentive for Displaced Workers: The Earnings Supplement Project," May. Ottawa, Ontario: Social Research and Demonstration Corporation. http://www.hrdc-drhc.gc.ca/sp-ps/arb-dgra/publications/research/1999docs/abr-99- 9e.shtml Blundell, Richard and Monica Costa Dias. 2000. "Evaluation Methods for Non-Experimental Data," Fiscal Studies 21, no. 4: 427-68. Blundell, Richard, Monica Costa Dias, Costas Meghir and John Van Reenen. 2001. "Evaluating the Employment Impact of a Mandatory Job Search Assistance Program," Working Paper no. 01/20, November. London, UK: Institute for Fiscal Studies. http://www.ifs.org.uk/workingpapers/wp0120.pdf Blundell, Richard and Costas Meghir. 2002. "Active Labour Market Policy vs. Employment Tax Credits: Lessons from recent UK reforms". University College London and Institute for Fiscal Studies. http://netec.mcc.ac.uk/WoPEc/data/Papers/hhsifauwp2002_001.html Boeri, Tito and Michael C. Burda. 1996. "Active Labor Market Policies, Job Matching and the Czech Miracle," European Economic Review 40, no. 3-5: 805-17. Bonnal, Liliane, Denis Fougère and Anne Sérandon. 1997. "Evaluating the Impact of French Employment Policies on Individual Labour Market Histories," Review of Economic Studies 64: 683-713. Brodaty, Thomas, Bruno Crépon and Denis Fougère. 2001. "Using Matching Estimators to Evaluate Alternative Youth Employment Programs: Evidence from France, 1986- 1988," in Michael Lechner and Friedhelm Pfeiffer (Eds.), Econometric evaluation of labour market policies (pp. 85-123). Heidelberg and New York: Physica-Verlag and Centre for European Economic Research. Bryson, Alex, Richard Dorsett and Susan Purdon. 2002. "The Use of Propensity Matching in the Evaluation of Active Labour Market Policies". Department of Work and Pensions. United Kingdom. 57 Burda, Michael C. and Martina Lubyova. 1995. "The Impact of Active Labor Market Policies: A Closer Look At The Czech and Slovak Republics," Discussion Paper no. 1102,. London: Centre for Economic Policy Research. Buttner, Thiess and Hedwig Prey. 1998. "Does Active Labor-Market Policy Affect Structural Unemployment? An Empirical Investigation for West German Regions, 1986 to 1993," Zeitschrift fur Wirtschafts- und Sozialwissenschaften 118, no. 3: 389-413. Calderon-Madrid, Angel and Belem Trejo. 2001. "The Impact of the Mexican Training Program for Unemployed Workers on Re-employment Dynamics and on Earnings." Research Network Papers. Inter-American Bank. Washington, D.C. http://econpapers.hhs.se/cpd/2002/86_Calderon.pdf Carling, Kenneth and Katarina Richardson. 2001. "The Relative Efficiency of Labor Market Programs: Swedish experience from the 1990s." Working Paper 2001:2. Swedish Office of Labour Market Policy Evaluation. Carling, Kenneth and Lena Gustafson. 1999. "Self-Employment Grants vs. Subsidized Employment: Is there a difference in the re-unemployment risk?." Swedish Office of Labor Market Policy Evaluation . http://netec.mcc.ac.uk/WoPEc/data/Papers/hhsifauwp1999_006.html Castro, Claudio de Moura. 1999. "Proyecto Joven. New Solutions and Some Surprises." EDU-110. Inter-American Development Bank. Washington D.C. Chan, William and Wing Suen. 2000. "An Evaluation of the Hong Kong Employees Retraining Programme," Asian Economic Journal 14, no. 3: 255-81. Cockx, Bart. 2000. "The Design of Active Labour Market Policies: Building in Effective Incentives," International Labour Review 139, no. 4: 457-80. Cockx, Bart and Isabelle Barboulat. 1999. "Vocational Training: Does it speed up the Transition Rate out of Unemployment?," IRES Discussion Paper no. 9932, December. Louvain-la-Neuve, Belgium: Université Catholique de Louvain. Cockx, Bart and Geert Ridder. 2001. "Social Employment of Welfare Recipients in Belgium: An Evaluation," Economic Journal 111, no. 470: 322-52. Cockx, Bart, Bruno Van der Linden and Adel Karaa. 1998. "Active Labour Market Policies and Job Tenure," Oxford Economic Papers 50, no. 4: 685-708. Dar, Amit and Indermit S. Gill. 1998. "Evaluating Retraining Programs in OECD Countries: Lessons Learned," World Bank Research Observer 13, no. 1: 79-101. 58 Dar, Amit and P. Zafiris Tzannatos. 1999. "Active Labor Market Programs: A Review of the Evidence from Evaluations," Social Protection Discussion Paper no. 9901, January. The World Bank. Washington, D.C. Decker, P., Robert Olsen and Lance Freeman. 2000. "Assisting Unemployment Insurance Claimants: The Long-Term Impacts of the Job Search Assistance Demonstration". Mathematica Policy Research. Washington, D.C. Dehejia, Rajeev H. and Sadek Wahba. 1998. "Propensity Score Matching Methods for Non- Experimental Causal Studies". Working Paper 6829. National Bureau of Economic Research. Boston. MA. Dehejia, Rajeev H. and Sadek Wahba. 1999. "Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs," Journal of the American Statistical Association 94, no. 448: 1053-62. Dockery, A. M. 1999. "Evaluating the Job Network," Australian Journal of Labour Economics 3, no. 2: 131-58. Eichler, Martin and Michael Lechner. 2000. "Some Econometric Evidence on the Effectiveness of Active Labor Market Programmes in East Germany". Working Paper Number. 318. Swiss Institute for International Economics and Applied Economic Research (SIW). Switzerland. Evans, Brendan. 1998. "Training Policy as a Palliative for Unemployment: The United Kingdom and Australia Compared," in Thomas Lange (Ed.) Unemployment in Theory and Practice (pp. 179-196). Cheltenham, U.K. and Northampton, Mass.: Elgar; distributed by American International Distribution Corporation Williston Vt. Fawcett, Caroline S. 2001. "Latin American Youth in Transition: A policy paper on youth unemployment in Latin America". Paper prepared for the "First Technical Meeting on Labor Market Issues". Panama City, Panama. http://www.eldis.org/static/DOC11803.htm Fay, Robert G. 1996. "Enhancing the Effectiveness of Active Labour Market Policies: Evidence from Programme Evaluations in OECD Countries." Labour Market and Social Policy Occasional Papers, No. 18. OECD, Paris. Fertig, Michael and Christoph M. Schmidt. 2000. "Discretionary Measures of Active Labor Market Policy: The German Employment Promotion Reform in Perspective," Schmollers Jahrbuch: Zeitschrift fur Wirtschafts-und Sozialwissenschaften/Journal of Applied Social Science Studies 120, no. 4: 537-65. Firth, David, Clive Payne and Joan Payne. 1999. "Efficacy of Programmes for the Unemployed: Discrete Time Modelling of Duration Data from a Matched 59 Comparison Study," Journal of the Royal Statistical Society, Series A 162, no. 1: 111- 20. Fitzenberger, Bernd and Hedwig Prey. 2000. "Evaluating Public Sector Sponsored Training in East Germany," Oxford Economic Papers 52, no. 3: 497-520. Fletcher, D. R. 1998. "The Effectiveness of Active Labour-Market Policies: Latest Evidence from the National Development Programme," Environment and Planning C: Government and Policy 16, no. 5: 573-89. Fougère, Denis, Francis Kramarz and Thierry Magnac. 2000. "Youth Employment Policies in France," European Economic Review 44, no. 4-6: 928-42. Freedman, Stephen. 2000. "The National Evaluation of Welfare-to-Work Strategies: Four- Year Impacts of Ten Programs on Employment Stability and Earnings Growth," Mimeo December. New York: Manpower Demostration Research Corporation. http://www.mdrc.org/publications/157/full.pdf Fretwell, David H., Jacob Benus, and Christopher J. O'Leary. 1999. "Evaluating the Impact of Active Labor Market Programs: Results of Cross Country Studies in Europe and Central Asia." Social Protection Discussion Paper no. 9915. The World Bank. Washington, DC: Galasso, Emanuela, Martin Ravallion, and Agustin Salvia. 2001. "Assisting the Transition from Workfare to Work: Argentina's Proempleo Experiment." Development Research Group, World Bank. http://poverty.worldbank.org/files/11255_Martin- PROEMPLEO-Evaluation.pdf Gerfin, Michael, Michael Lechner and Heidi Steiger. 2002. "Does Subsidized Temporary Employment Get the Unemployed Back to Work?." Discussion Paper No. 606. Centre for Economic Policy Research. London. http://netec.mcc.ac.uk/WoPEc/data/Papers/cprceprdp3669.html Godfrey, Martin. 2003. "Youth Employment Policy in Developing and Transition Countries ­ Prevention as well as Cure." Final draft prepared for Social Protection Unit, World Bank. Washington, D.C. Gonzala, Maria Teresa and Jan Saarela. 2001. "An Evaluation of the Management of the Finnish Employment Service," Applied Economics 33, no. 13: 1717-25. Graves, Frank and Benoît Gauthier. 2000. "Evaluation of the Self-Employment Assistance Program". Human Resources Development Canada. http://www11.hrdc-drhc.gc.ca/pls/edd/SEAP_brf.shtml 60 Gritz, Mark and Terry Johnson. 2001. "National Job Corps Study: Assessing Program Effects on Earnings for Students Achieving Key Program Milestones". Impact evaluation submitted to U.S. Department of Labor. Battelle Memorial Institute. Seattle, Washington. http://wdr.doleta.gov/opr/fulltext/document.cfm?docn=6158 Grubb, W. Norton and Paul Ryan. 1999. "The roles of evaluation for vocational education and training: Plain talk on the field of dreams." Geneva: International Labour Office; London: Kogan Page; distributed by ILO Publications Center Waldorf Md. Grupo de Analisis para el Desarrollo. 2001. "Una medicion del impacto del Programa de Capacitacion Laboral Juvenil PROJoven." Lima, Peru. http://pubweb.northwestern.edu/~hrn663/projoven.pdf Heckman, James J. 1999. "Accounting for Heterogeneity, Diversity and General Equilibrium in Evaluating Social Programs," Working Paper no. 7230. National Bureau of Economic Research. Boston, MA. Heckman, James J., Neil Hohmann and Jeffrey Smith. 2000. "Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment," Quarterly Journal of Economics 115, no. 2: 651-94. Heckman, James J., Robert J. Lalonde and Jeffrey A. Smith. 1999. "The Economics and Econometrics of Active Labor Market Programs," in Orley Ashenfelter and David Card (Eds.), Handbook of labor economics, vol. 3A (pp. 1865-2097). Handbooks in Economics, vol. 5. Amsterdam; New York and Oxford: Elsevier Science North- Holland. Heckman, James J. and Jeffrey A. Smith. 1999. "The Pre-programme Earnings Dip and the Determinants of Participation in a Social Programme Implications for Simple Programme Evaluation Strategies," Economic Journal 109, no. 457: 313-48. Heckman, James J., Justin L. Tobias and Edward Vytlacil. 2001. "Four Parameters of Interest in the Evaluation of Social Programs," Southern Economic Journal 68, no. 2: 210-23. Hendra, Richard, Charles Michalopoulos and Dan Bloom. 2001. "Three-Year Impacts of Connecticut's Jobs First Welfare Reform Initiative," Mimeo April. Manpower Demostration Research Corporation. New York. http://www.mdrc.org/Reports2001/CT-JobsFirst- 3YrImpacts/CT3YrImpactsExSum.htm Hollenbeck, Kevin. 2003. "Net Impact Estimates of the Workforce Development System in Washington State". Upjohn Institute Staff Working Paper No. 03-92 http://netec.mcc.ac.uk/WoPEc/data/Papers/upjweupjo03-92.html 61 Hui, Shek-Wai and Jeffrey Smith. 2002. "The Labor Market Impacts of Adult Education and Training in Canada". Human Resources Development Canada. Hujer, R. and M. Wellner. 2000. "The Effects of Public Sponsored Training on Individual Employment Performance in East Germany". Working Paper, Frankfurt: J.W.Goethe University. Human Resources Development Canada. 1999."Summative Evaluation of Youth Service Canada". http://www11.hrdc-drhc.gc.ca/pls/edd/hrdc.main Human Resources Development Canada. 1995. "Evaluation of the Employability Improvement Program," Mimeo December Evaluation and Data Development. http://www11.hrdc-drhc.gc.ca/pls/edd/EIP_brf.shtml Human Resources Development Canada. 1996. "A Review of Programs for Integrating Social Assistance Recipients into the Workforce," Mimeo December Evaluation and Data Development. http://www11.hrdc-drhc.gc.ca/pls/edd/SARW.shtml Impaq International. 2001. "Bosnia and Herzegovina: Impact of the Emergency Demobilization and Reintegration Project." Prepared for the U.S. Department of Labor. International Labour Organization. 2003. "Youth training and employment". Inter-American Research and Documentation Centre on Vocational Training (Cinterfor/ILO). Geneva. Jacob, Louis and Ian Petta.2000. "Measuring the Effect of Public Labor Exchange Referrals and Placements in Washington and Oregon." Westat. Maryland. http://wdr.doleta.gov/owsdrr/00-6/00-6.pdf Jalan, Jyotsna and Martin Ravallion. 2002. "Estimating the Benefit Incidence of an Antipoverty Program by Propensity-Score Matching," Journal of Business and Economic Statistics. http://fmwww.bc.edu/RePEc/es2000/0873.pdf Kang, Soon-Hie and Byung Hee Lee. 2001. "Evaluating the Training for the Unemployed in Korea"; in: Labor Market Reforms in Korea: Policy Options for the Future. Edited by: Funkoo Park, Young-bum Park, Gordon Betcherman and Amit Dar. World Bank and Korea LAbor Institute. Kavanagh, Catherine. 1998. "A Review of the Role of Active Labour Market Policies in Ireland," in Charles M. A. Clark and Catherine Kavanagh (Eds.), Unemployment in Ireland: Alternative perspectives (pp. 43-67). Aldershot, U.K.; Brookfield, Vt. and Sydney: Ashgate. 62 Kluve, Jochen and Christoph M. Schmidt. 2002. "Can Training and Employment Subsidies Combat European Unemployment?" Economic Policy. Vol. 35. October. Pp. 411- 448. Kluve, Jochen, Hartmut Lehmann and Christoph M. Schmidt. 1999. "Active Labor Market Policies in Poland: Human Capital Enhancement, Stigmatization, or Benefit Churning?," Journal of Comparative Economics 27, no. 1: 61-89. Kluve, Jochen, Hartmut Lehmann and Christoph M. Schmidt. 2002. "Disentangling Treatment Effects of Polish Active Labour Market Policies: Evidence From Matched Samples," Discussion Paper no. 3298, April. London, UK: Centre for Economic Policy Research. Kosanovich, William and Heather Fleck. 2001. "Comprehensive Assessment of Self- Employment Assistance Programs". DTI Associates, Inc. Prepared for: U.S. Department of Labor. Office of Workforce Security. Kramer, Fredrica D., Demetra Smith Nightingale, John Trutko, Shayne Spaulding and Burt S. Barnow. 2002. "Faith-Based Organizations Providing Employment and Training Services: A Preliminary Exploration," Mimeo February. The Urban Institute, Washington, DC. Kraus, Florian, Patrick Puhani and Viktor Steiner. 1999. "Employment Effects of Publicly Financed Training Programs-The East German Experience," Jahrbucher fur Nationalokonomie und Statistik 219, no. 1-2: 216-48. LaLonde, R.J. 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data. American Economic Review Vol. 76 (4). Lange, Thomas. 1998. Unemployment in theory and practice. Cheltenham, U.K. and Northampton, Mass.: Elgar; distributed by American International Distribution Corporation Williston Vt. Lange, Thomas and J. R. Shackleton. 1998. "Active Labour Market Policies: A Critical Assessment," in Thomas Lange (Ed.) Unemployment in theory and practice (pp. 145- 63). Cheltenham, U.K. and Northampton, Mass.: Elgar; distributed by American International Distribution Corporation Williston Vt. Larson, Laura. 2000. "Evaluation of Swedish youth labour market programmes". Uppsala University & Office of Labour Market Policy Evaluation. http://ideas.repec.org/p/hhs/uunewp/2000_006.html Lechner, Michael. 1998a. "Continuous Off-the-Job Training in East Germany after Unification: Preliminary Results of an Evaluation of the Short-Run Effects for Individual Workers," in Horst Brezinski, Egon Franck and Michael Fritsch (Eds.), 63 The microeconomics of transformation and growth (pp. 205-30). Cheltenham, U.K. and Northampton, Mass.: Elgar; distributed by American International Distribution Corporation Williston Vt. Lechner, Michael. 1998b. Training the East German labour force: Microeconometric evaluations of continuous vocational training after unification. Studies in Contemporary Economics. Heidelberg: Physica. Lechner, Michael. 1999a. "Earnings and Employment Effects of Continuous Off-the-Job Training in East Germany After Unification," Journal of Business and Economic Statistics 17, no. 1: 74-90. Lechner, Michael. 1999b. "The Effects of Enterprise-Related Training in East Germany on Individual Employment and Earnings," Annales d'Economie et de Statistique 0, no. 55-56: 97-128. Lechner, Michael. 2000. "An Evaluation of Public-Sector-Sponsored Continuous Vocational Training Programs in East Germany," Journal of Human Resources 35, no. 2: 347-75. Lechner, Michael and Friedhelm Pfeiffer (Eds.). 2001. Econometric evaluation of labour market policies. Heidelberg and New York: Physica-Verlag and Centre for European Economic Research. Lise, Jeremy, Shannon Seitz and Jeffrey Smith. 2003. "Equilibrium Policy experiments and the Evaluation of Social Programs.". Queen's University. Kingston. http://ideas.repec.org/p/iza/izadps/dp758.html Lubyova, Martina and Jan C. van Ours. 1999. "Effects of Active Labor Market Programs on the Transition Rate from Unemployment into Regular Jobs in the Slovak Republic," Journal of Comparative Economics 27, no. 1: 90-112. Magnac, Thierry. 2000. "Subsidised Training and Youth Employment: Distinguishing Unobserved Heterogeneity from State Dependence in Labour Market Histories," Economic Journal 110, no. 466: 805-37. http://ideas.repec.org/a/ecj/econjl/v110y2000i466p805-37.html Martin, John P. 2000. "What Works among Active Labour Market Policies: Evidence from OECD Countries' Experiences," OECD Economic Studies, no. 30: 79-113. http://ideas.repec.org/p/rba/rbaacv/1998-17.html Martin, John P. and David Grubb.2001. "What works and for whom: A Review of OECD countries' experiences with active labour market policies." Office of Labour Market Policy Evaluation. Working Paper 2001:14. http://netec.mcc.ac.uk/WoPEc/data/Papers/hhsifauwp2001_014.html 64 Marx, Ive. 2001. "Job Subsidies and Cuts in Employers' Social Security Contributions: The Verdict of Empirical Evaluation Studies," International Labour Review 140, no. 1: 69-83. Medina, Carlos and Jairo Nuñez. 2001. "The Impact of Public and Private Job Training in Colombia." Research Network Paper. Inter-American Bank. Washington, D.C. http://www.iadb.org/res/laresnetwork/projects/pr201finaldraft.pdf Michalopoulos, Charles, Christine Schwartz and Diana Adams-Ciardullo. 2001. "National Evaluation of Welfare-to-Work Strategies: What Works Best for Whom: Impacts of 20 Welfare-to-Work Programs by Subgroup," Mimeo January. Manpower Demonstration Research Corporation. New York. Michaelopoulos, Charles et al.. 2002. "Making Work Pay: Final Report on the Self- Sufficiency Project for Long-Term Welfare Recipients." Social Research and Demonstration Corporation. Canada. http://www.mdrc.org/publications/46/abstract.html Miller, Cynthia, Johannes M. Bros, Kristin E. Porter and others. 2003. "Working with Disadvantaged Youth. Thirty-Month Findings from the Evaluation of the Center for Employment Training Replication Sites". Manpower Demonstration Research Corporation. http://www.mdrc.org/publications/343/overview.html Naranjo Silva, Alicia. 2002. "Capacitacion y formacion profesional para jovenes en Uruguay." CINTERFOR/PNUD. Montevideo. OECD. 2002. Employment Outlook. Paris. OECD. 2003. Employment Outlook. Paris. Olga, Kupets. 2000. "The Impact of Active Labor Market Policies on the Outflows from Unemployment to regular jobs in Ukraine." Thesis for the degree of Master of Arts in Economics. Nation University. Kiev Mohyla Academy. O'Higgins, Niall. 2001. Youth unemployment and employment policy: A global perspective. Geneva: International Labour Office; distributed by ILO Publications Center Waldorf Md. O'Higgins, Niall. 2003. "Trends in the Youth Labour Market in Developing and Transition Countries." Draft prepared for the Social Protection Unit, World Bank. Washington, D.C. O'Leary, Christopher, Paul Decker and Stephen Wandner. 2003. "Cost-effectiveness of targeted Reemployment Bonuses". Upjohn Institute Staff Working Paper. 65 O'Leary, Christopher, Alena Nesporova and Alexander Samorodov. 2001. Manual on Evaluation of Labour Market Policies in Transition Economies. International Labour Office. Geneva. O'Leary, Christopher J. 1999. "Promoting Self Employment Among the Unemployed in Hungary and Poland," Working Paper no. 99-055, February. W.E. Upjohn Institute for Employment Research. Kalamazoo, MI. http://netec.mcc.ac.uk/WoPEc/data/Papers/upjweupjo99-55.html O'Leary, Christopher J., Piotr Kolodziejczyk and Gyorgy Lazar. 1998. "The Net Impact of Active Labour Programmes in Hungary and Poland," International Labour Review 137, no. 3: 321-46. Olsen, Robert B. and Paul T. Decker. 2001. "Testing Different Methods of Estimating the Impacts of Worker Profiling and Reemployment Services Systems," Mimeo June. Mathematica Policy Research, Inc. Washington, DC. http://wdr.doleta.gov/opr/fulltext/document.cfm?docn=6154 Park, Funkoo, Young-bum Park, Gordon Betcherman and Amit Dar (Eds.). 2001. Labor Market Reforms in Korea: Policy Options for the Future. The World Bank and Korea Labor Institute. Seoul and Washington, DC. Park, Norm, Bob Power, W. Craig Riddell and Ging Wong. 1996. "An Assessment of the Impact of Government-Sponsored Training". Canadian Journal of Economics. Volume 29, Special Issue: Part 1 (Apr., 1996), S93-S98. Payne, Joan. 2000. "Evaluating Training Programmes for the Long-Term Unemployed". Policy Studies Institute. Research Discussion Paper 1. London. Pfeiffer, Friedhelm and Frank Reize. 2000. "Business Start-ups by the Unemployed--An Econometric Analysis Based on Firm Data," Labour Economics 7, no. 5: 629-63. Prey, Hedwig. 2000. "Evaluation of Training Programs in St Gallen, Switzerland," Schweizerische Zeitschrift fur Volkswirtschaft und Statistik/Swiss Journal of Economics and Statistics 136, no. 3: 417-32. Puhani, Patrick A. 1998. "Advantage Through Training? A Microeconometric Evaluation of the Employment Effects of Active Labour Market Programmes in Poland," Discussion Paper no. 2000,. London: Centre for Economic Policy Research. http://netec.mcc.ac.uk/WoPEc/data/Papers/cprceprdp2000.html Puhani, Patrick A. 1999. "Evaluating active labour market policies: Empirical evidence for Poland during transition". ZEW Economic Studies, vol. 5. Centre for European Economic Research (ZEW). Heidelberg. 66 Raaum, Oddbjorn and Hege Torp. 2000. "Labour Market Training in Norway - Effect on Earnings". Department of Economics, University of Bergen. Oslo, Norway. Ravallion, Martin. 2001. "The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation," World Bank Economic Review 15, no. 1: 115-40. Ravallion, Martin, Emanuela Galasso, Teodoro Lazo, and Ernesto Philip. "Do Workfare Participants Recover Quickly from Retrenchment?" Development Research Group, World Bank. http://www.econ.yale.edu/seminars/labor/lap02/ravellion2-020329.pdf Riccio, James A. and Howard S. Bloom. 2001. "Extending the Reach of Randomized Social Experiments: New Directions in Evaluations of American Welfare-to-Work and Employment Initiatives," MDRC Working Papers on Research Methodology October. New York: Manpower Demonstration Research Corporation. Robinson, Peter. 1995. "The Decline of the Swedish Model and the Limits to the Active Labour Market Policy," Discussion Paper no. 259. London School of Economics, Centre for Economic Performance. London. Rose, Nancy E. 1999. "Jobs for Whom? Employment Policy in the United States and Western Europe," Journal of Economic Issues 33, no. 2: 453-60. Roy, Arun S. and Ging Wong. 2000. "Direct Job Creation Programs: Evaluation Lessons on Cost-Effectiveness," Canadian Public Policy 26, no. 2: 157-69. http://www11.hrdc-drhc.gc.ca/pls/edd/DJC.shtml Ryan, Paul. 2000. "Publicly Funded Training for Unemployed Adults: Germany, the UK and Korea"; in: Labor Market Reforms in Korea: Policy Options for the Future. Edited by Funkoo Park, Young-bum Park, Gordon Betcherman, Amit Dar. World Bank. Korea Labor Institute. Ruppert, Elizabeth. 1999. "The Algerian Retrenchment System: A Financial and Economic Evaluation," World Bank Economic Review 13, no. 1: 155-83. Santiago Consultores Asociados. 1998. "Evaluacion Ex Post Programa Chile Joven Fase II". Chile. Schmid, Gunther. 1995. "Institutional Incentives to Prevent Unemployment: Unemployment Insurance and Active Labor Market Policy in a Comparative Perspective," Journal of Socio-Economics 24, no. 1: 51-103. Sianesi, Barbara. 2001. "Differential Effects of Swedish Active Labour Market Programmes for Unemployed Adults during the 1990s". Institute for Labour Market Policy Evaluation. http://netec.mcc.ac.uk/WoPEc/data/Papers/hhsifauwp2002_005.html 67 Smith, Jeffrey. 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Schweizerische Zeitschrift fur Volkswirtschaft und Statistik/Swiss Journal of Economics and Statistics 136, no. 3: 247-68. Smith, Jeffrey A. 2000. "Evaluating Active Labor Market Policies: Lessons from North America". University of Western Ontario and NBER. http://www.glue.umd.edu/~jsmithz/papers/nu081500.pdf Smith, Jeffrey A. and Petra E. Todd. 2001. "Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods," American Economic Review 91, no. 2: 112-18. Storto, Laura, Gayle Hamilton, Christine Schwartz and Susan Scrivener. 2000. "National Evaluation of Welfare-to-Work Strategies: Oklahoma City's ET & E Program: Two- Year Implementation, Participation, Cost, and Impact Findings," Mimeo August. Manpower Demonstration Research Corporation. New York. Stromback, Thorsten, A. M. Dockery and Wiwi Ying. 1999. "Labour Market Programs and Labour Force Status," Australian Bulletin of Labour 25, no. 2: 159-78. Subbarao, Kalanidhi. 2003. "Systemic Shocks and Social Protection: Role and Effectiveness of Public Works Programs." Social Protection Discussion Paper No. 0302. The World Bank. Washington, DC. Terrell, Katherine and Vit Sorm. 1999. "Labor Market Policies and Unemployment in the Czech Republic," Journal of Comparative Economics 27, no. 1: 33-60. Van Reenen, John. 2003. "Active Labour Market Policies and the British New Deal for the Young Unemployed in Context". NBER Working Paper Series. Working Paper 9576. Cambridge, Massachusetts. Vodopivec, Milan. 1999. "Does the Slovenian Public Work Program Increase Participants' Chances to Find a Job?," Journal of Comparative Economics 27, no. 1: 113-30. Walsh, K., M. Kotzeva, E. Dolle and R. Dorenbos. 2001. "Evaluation of the Net Impact of Active Labour Market Programs in Bulgaria". Ministry of Labour and Social Policy, Bulgaria. Weller, Sally and Michael Webber. 1999. "Re-employment after Retrenchment: Evidence from the TCF Industry Study," Australian Economic Review 32, no. 2: 105-29. Woltermann, Silke. 2002. "Job-Search Methods and Labor Market Transitions." University of Gottingen. Denmark. http://econpapers.hhs.se/paper/wpawuwpla/0310001.htm 68 Wondon, Quentin and Mari Minowa. 1999. "Training for the Urban Unemployed: A Reevaluation of Mexico's PROBECAT. World Bank", Government Programs and Poverty in Mexico, Report No. 19214-ME, Vol. II. World Bank. 2002a. "Impact Evaluation: Techniques for Evaluating Active Labor Market Programs." Employment Policy Primer Series No. 2. Social Protection Unit. World Bank. 2002b. "Evaluation of Active Labor Programs in Macedonia. Final Report." Social Reform and Technical Assistance Project. IDA-2722-MK. World Bank. World Bank. 2003. "Republic of Tunisia Employment Strategy." Human Development, Middle East and North Africa Region, World Bank. 69 Annex 1: Expenditures on Active Labor Market Programs in OECD Countries Distribution of Expenditures on ALMPs (selected OECD countries) 2000-2001 Public Others Employment 16% Services 24% Micro-enterprises 2% Job subsidies 21% Training 37% Source: OECD. Employment Outlook 2002. Public expenditure in Public Employment Services and Administration (Percentage of GDP) United States Korea Greece Czech Republic Spain Switzerland Portugal New Zealand Hungary Norway Finland Denmark United Kingdom Austria Canada Belgium France Japan Australia Sweden Germany Netherlands 0 0.05 0.1 0.15 0.2 0.25 0.3 Source: OECD Employment Outlook 2002. 70 Public expenditure in Labor Market Training (Percentage of GDP) Poland Czech Republic Australia Mexico Japan Italy Norway Hungary Switzerland Korea Spain Portugal New Zealand Canada Austria Belgium France Finland Sweden Netherlands Germany United States United Kingdom Denmark 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Source: OECD Employment Outlook 2002. Public expenditure in youth training 2000-2001 (Percentage of GDP) Germany Canada Austria Netherlands Poland New Zealand Australia Finland United Kingdom Portugal France Italy 0 0.05 0.1 0.15 0.2 0.25 Source: OECD Employment Outlook 2002. 71 Public expenditure in subsidies to regular employment in the private sector 2000-2001 (Percentage of GDP) United Kingdom Portugal Norway Korea Australia Poland Denmark Germany Switzerland Czech Republic New Zealand Netherlands Austria Sweden Hungary Finland France Italy Spain Belgium 0 0.05 0.1 0.15 0.2 0.25 0.3 Source: OECD Employment Outlook 2002. Public expenditure in direct job creation programs 2000-2001* (Percentage of GDP) United States New Zealand Poland Mexico Canada Austria Czech Republic Portugal Italy Spain Switzerland Sweden Australia Finland Korea Denmark France Hungary Germany Netherlands Belgium 0 0.1 0.2 0.3 0.4 0.5 0.6 *Include public or non-profit jobs. Source: OECD Employment Outlook 2002. 72 Public expenditure in support of unemployed presons starting enterprises 2000-2001 (Percentage of GDP) Poland Korea Hungary Czech Republic Canada Australia Austria Portugal New Zealand Finland Sweden Italy Germany Spain 0 0.01 0.02 0.03 0.04 0.05 0.06 Source: OECD Employment Outlook 2002. 73 Annex 2: Summary of Recent Evaluations of Active Labor Market Programs Job Search Assistance / Public Employment Services Labor Market Issue Intervention Design Type of Evaluation Result Comments OECD / Developed countries Australia. Job clubs provide Logit on the The program has no significant The period examined is not Job Clubs jobseekers with probability of effects on either probability of discussed, so it is difficult to Period 1993- 1996 instruction in job search staying in the labor staying in the labor force or establish how long after [Stromback et al., 1999] techniques and support. force and on finding a job. participation that status is Also aimed to improve employment. observed. employment prospects by increasing jobseeker's self esteem, confidence and job search efforts. Great Britain. After 6 months of Quasi-experimental. Young unemployed men are The New Deal treatment New Deal for the unemployment, 18-24 Econometric about 20% more likely per effect comprises the effects Young Unemployed. years olds enter a combined with period to gain jobs as a result of of both the job Significant youth "Gateway" period where matching. the New Deal. Part of this effect assistance/monitoring unemployment, they are given extensive is due to subsidized jobs, and element of New Deal and especially among lower job search assistance. If part a pure "Gateway" element the wage subsidy element. educated and less they are unable to obtain (enhanced job search), at least The small sample size for skilled. unsubsidized job, then one fifth of the total effect. The women does not allow to Period 1998- 2002 they can enter one of four cost benefit analysis suggests obtain precise estimates for [Van Reenen, 2003] New Deal options: job that the program is worth women. Most estimators for subsidy, training, continuing. women are statistically environmental task force insignificant. or voluntary work. United States. The Job Search AssistanceExperimental. JSA treatments present uneven Costs of the program were Job Search Assistance Demonstration had three Regression Analysis impacts on employment and between $97 and $286 per Demonstration. service strategies: with follow-up earnings. claimant. The rates of return Re-employment -Structured Job Search survey data. The SJSA group in D.C. are in general negative, problems of Assistance (SJSA): generally had higher earnings because the estimated permanently laid-off orientation, testing, and the differences tend to be reduction in unemployment workers have received workshop, assessment statistically significant. The insurance payments caused national attention since interview). estimated impacts of SJSA on by treatments were not large 1980s. -Individualized Job Search earnings in Florida and the enough to fully compensate Washington DC and Assistance (IJSA): estimated impacts of IJSA and for the costs of services. Florida. assigned claimants to IJSA+ on earnings in both states However, from society's Period 1995-1996 services based on their tended to be smaller and not perspective as a whole the [Decker, 2000] assessed needs. statistically significant in most program has positive rates - Individualized Job cases. of return (from 826% Search Assistance with JSA treatments were not to391%) in Washington DC. training IJSA+): Identical generally cost-effective. In Florida the rates of return to the second one plus the are negative by large, inclusion of training. between ­416% to ­59%. United States. Direct placement services Quasi experimental. Job seekers with strong work Results for a pilot sample. It Public Labor provided by public labor Complementing the records experienced a 7.2 week is not clear that the results Exchanges. exchanges to job seekers. evaluation with a reduction in their duration of are representative for all Washington and Oregon mail survey. unemployment compared to placements. Period 1987 ­1998 control group and reduces the [Jacobson and Petta, time for placing job seekers with 2000] spotty work experience in 3.4 weeks. The increase of earnings was $1,872 and $684 for job seekers with strong and spotty work records, respectively. The benefit-cost ratio was 1.8 for the sample studied. 74 Job Search Assistance / Public Employment Services Labor Market Issue Intervention Design Type of Evaluation Result Comments Transition Countries Intervention Design Type of Evaluation Result Comments Czech Republic. District labor offices act asCompeting risk Program reduced [Terrell and Storm, job brokers. This service hazard model. The unemployment spells for 1999] is open to all registered study evaluates groups that tend to have unemployed. impact of the longer unemployment program considering spells. either job found through labor offices or by individual. Macedonia. Job counseling, labor Quasi-experimental..Positive and significant Counseling market information, job Logistical regressionimpact on its participants' Period 1994-2000 search. analysis. opportunities. 14% of [World Bank, 2002b] participants found a job because of the program. Romania. Clients eligible for this Quasi-experimental. Positive impact on current Employment and service were offered: job Propensity Score employment (7% of Relocation program. and social counseling, Matching. higher likelihood of being Period 1999-2001 labor market information, employed) . The program [Benus and Rodriguez-job search assistance, job reduced the length of the Planas, 2002] placement services and current unemployment relocation assistance. spell. Positive impact on current earnings. Strong positive impact for males and a zero impact for females. Developing countries Intervention Design Type of Evaluation Result Comments Brazil. SINE agencies provide Quasi-experimental. Public employment A previous evaluation of National System forassistance in job Multinomial logit service shows negative Brazilian Ministry of Labor Employment (SINE).placement and job models. Data from a effects for the transition (not available) has shown Job placement formatching to the pool of Monthly into self-employment or very poor performance in unemployed workers. unemployed. Employment inactivity. Most of the job most SINE agencies. Period 1999 Surveys. searchers in Brazil recur [Woltermann, 2002] to channels that involve directly asking an employer or friends and family. Search channels that provide collection of information on vacancies are only used by a small share of workers. Uruguay. No information provided. No information Employment impact is The study summarizes some Labor Market provided. linked to the youth findings from Latin information services. education profile; American programs. The [Fawcett, 2001] unskilled workers with primary evaluation source low education have not found. limited amounts of "information", higher educated youth have more information. Greatest gains to higher educated youth workforce in formal sector. Reduces search time (opportunity cost) to entrant and hiring cost to employer. 75 Training (LT Unemployed) Labor Market Issue Intervention Design Type of Evaluation Result Comments OECD / Developed countries Australia. Consist of both skills Logit on the For both males and females Period 1994-95 training and employment probability of staying there is a significant positive [Stromback et al, 1999]training. The latter included in the labor force and effect of employment training, work experience, especially on employment. while there is no effect of skills aimed at long-term training. unemployed. Belgium. Vocational classroom Minimum Chi-Square Training found to considerably Not possible to determine Persistent training, mostly consisting estimation to transform speed up the transition rate out among the different types unemployment inof basic skills required in the transition model of unemployment. However, if of training. According to Belgium, who has oneparticular vocations. between employment the worker is still unemployed the authors, program of the highest shares ofParticipation is voluntary and unemployment intoafter participation in the induces non-participants long-term unemployedand any unemployed can a linear regression program, the rate at which to postpone their exit out in OECD. apply. Participants are then model. The authors unemployment is left is 62% of unemployment. Period 1989-1993 selected based on their analyze only spells of higher than in the absence of [Cockx and Bardoulat, "motivation". Median lengthunemployment with 6 training. 1999] of training is 2 months. moths or more of The returns to training are duration. rapidly diminishing. Belgium. There are both classroom Hazard model of lengthOn-the-job training lengthens [Cockx et al., 1998] training organized by public of employment with job tenure significantly, institutions and on-the-job corrections for although the magnitude is training in private firms. Theselection and bias from sensitive to specifications. The length of training and job sampling method. pure training program has guaranty depends on the positive but insignificant effect program (the latter only for on the length of the job tenure. on-the-job training participants). Canada. Training programs for Quasi-experimental to Impacts for skill shortages, The study summarizes Period 1988-1991 unemployment insurance estimate program feepayer and job entry are all key findings of an [Park et. al., 1996] claimants. impacts on both large in size and highly evaluation of five -Feepayer option: claimants employment and significant, while those for government-sponsored are exempted from having tohourly wages. part-time training and Job training programs in search for a job while taking Difference-in- Development are positive but Canada. No explanations training. difference estimators. generally not significantly provided to explain the -Part-time training. different from zero. differences in the impacts -Job Development for long of the programs. term unemployed -Job Entry for women re- entering labor force after an absence of at least three years. -Skill Shortages program. China (Hong Kong). Originally designed for Fixed effects Core training is found to have Industrial restructuring displaced workers, but later estimation of earnings. no significant effect on in Hong Kong with expanded significantly in Random effects probit earnings. Skills training is many manufacturing coverage. Two types of of employment status. generally associated with lower firms moving to courses: core and skill. Core earnings, although the effect is mainland China. courses are typically one- insignificant. The effect of Period 1994 -1999 week aimed at building self- participation in skills training [Chan and Suen, 2000] confidence and enhancing is uniformly negative and knowledge of job market significant on employment and and interviewing techniques. earnings. Skill courses last more than The improvement on earnings a month either full- or part- is $400 less than the time; most are in general comparison group. skills (language, computers). 76 Training (LT Unemployed) Labor Market Issue Intervention Design Type of Evaluation Result Comments Great Britain. ET and EA are programs Matching using a Over half of participants Matching done on sex, age, Employment Training for adults who had been discrete time hazard in ET received a formal local geographical area and (ET) and Employment unemployed and claiming model of event qualification: on average prior unemployment. Action (EA). benefits for 6 months or histories. 6 months' job training Persistent long-term more. and work experience. EA unemployment in had focus on work Britain during the last experience and was much two decades. smaller than ET. Period 1998-1999 Positive effect of [Firth, 1999] participation in ET on the hazard rate of entering a job, both while in ET and afterwards. The effect of EA was not significant. Great Britain. TfW included: employer Quasi-experimental. TfW had a positive Training for Work placements, project Matching. impact on employment (TfW) Program for the placements ­trainee prospects. In the year and long-term unemployed. worked with a voluntary a half after leaving TfW, Period 1995-97 organization- and full- participants spent on [Payne, 2000] time off-the-job training. average around one extra month in seven in work, in comparison to the control group. Most of the employment gains associated with the program came from an increased chance of getting a full-time job as an employee. Participation did not affect hourly wage rates. Germany. Three types of public Hazard rate models Training between January Uses Labor Market Monitor Large increase in training exist: Short with accounting for 1989 and August 1992 (LMM) data. There is no unemployment in East training courses (6 selection bias and had a negative impact on separate analysis of the three Germany after weeks), continuous unobservable the chance of finding different types of training, unification with West training in current heterogeneity. stable employment. although the expected return Germany. Period 1989- occupation (7-12 months) Employment after Training given between must be very different. 1994. and retraining (12-24 training is divided September 1992 and [Kraus et al., 1999] months). into stable and November 1994, when unstable. the institutional structure for the programs was in place, increased the probability of finding stable employment for both men and women. Germany. CTRT consists of three Matching, using the The effect immediately The study includes both Continuous vocational types, although only exogeneity of the after training is additional employed and unemployed training and retraining continuous training within unification as unemployment, although in the sample since both (CTRT). the current occupation andidentifying this effect disappears afterwere able to receive the Large increase in retraining to a new condition. some months. Overall, training, although the longer unemployment in East occupation are analyzed there is no significant somebody had been Germany after (they account for 90 per effect of participation in unemployed the better unification. cent of all entries). The training, which is thought chance they had for Period 1990- 1996 training is 99% classroom to be due to the reduced receiving training. [Lechner, 2000] training. Average duration search efforts during the of training is 12 months. training. 77 Training (LT Unemployed) Labor Market Issue Intervention Design Type of Evaluation Result Comments Germany. Two training programs: Quasi-experimental. -Future unemployment The study is an update of Large increase in - On-the job-training Matching. probabilities: No significant previous evaluations of the unemployment in East (OJT). Several months of effects of OJT. CT/RT same author (Lechner). Germany after formal training with an presents a short-term negative unification. external provider. effect. (Same programs - Training subsidized by -Monthly Earnings: Positive evaluated above but the labor offices, includes effects from OJT in the with different data). continues training (CT) second year after completion. Period 1991-1994 and retraining for a new The earnings are higher for [Eichler and Lechner, occupation (RT). To those not in highly qualified 2000] participate in CTRT an and/or managerial job individual must already positions. No significant hold a profession. effects on earnings of CTRT participants. Germany. Training consisting of Quasi-experimental. Findings mostly suggest The difference-in-difference is Large increase in either continuous Evaluates the impactpositive, though only partiallythe difference between the unemployment in East vocational training or of public sector significant, long-run effects estimated parameters of Germany after retraining for unemployed.training programs onof training on employment or dummies indicating unification. The study looks only at individual wages. However, when doing participation in the past and Period 1990-94 program participants who employment sensitivity tests the positive participation in the future. [Fitzenberger and Prey, have received financial probabilities and employment effects appear 2000] support in addition to the wages. quite stable across estimated training. models. Strong state dependence in employment: the probability of being employed in the next period is considerably higher when employed in this period. Germany. No details of training Quasi-experimental. No significant effects. No Public sector sponsored characteristics are Matching. evidence that participation in training. provided. public vocational training East Germany after significantly decreases unification. unemployment duration or Period 1990-1993 increases employment [Hujer and Wellner, duration. 2000] Korea. Anyone who had worked Quasi-experimental. Re-employment rates not No details of training Promote re-employment at employment insurance Matched pairs significantly different characteristics are provided. for the unemployed. applicable establishments comparisons and between participants and non- Period 1998-1999 is eligible for re- hazard model. participants. However, when [Kang and Lee, 2001] employment training for considering the point from the unemployed. which training participants actively begin to seek jobs, they are more successful in finding jobs. Norway. LMT covers about 40% ofQuasi-experimental. All models report positive Use of an internal comparison Labor Market Training all ALMP participants. Use of different effects of being offered group of eligible persons who program (LMT) The program is organized models to estimate training, though not always applied for training, but were targeted at unemployed as off-the job courses. impacts. significantly different from rejected by the administrators adults. Most courses are short, zero. Training effects with due to limited capacity of the Period 1989-1994 from 5 to 20 weeks. linear control models are program. Authors mention that [Raaum and Torp, Courses are free of about 15% to 20% of average though the effects of training 2000] charge. post-training earnings. on earnings are quite large, the effects on permanent earnings are likely to considerably lower. 78 Training (LT Unemployed) Labor Market Issue Intervention Design Type of Evaluation Result Comments Spain. Specific skills training in Quasi-experimental. Training courses have National Plan of various sectors. Youth, Estimates of positive effect reducing the Training and Re- women, disabled workers Kaplan-Meier for probability of remaining employment. Period and immigrants eligible. "survival analysis". unemployed, though are not 2000-2001 so effective as continuing in [Arellano, 2002] formal education. Groups that benefit the least are women, older workers and disabled workers. Switzerland. Three courses are Matching with Both the general basic and the Rapidly rising analyzed. The basic subsequent probit computer courses have unemployment. general training (for estimation of the significant negative impacts Period 1992- 1997 newly unemployed), effect of on the probability of being [Prey, 2000] which lasts about 10 days. participation on employed, while the German A German language for chance of being course shows large and foreigners course of 10-12 employed. significant positive effects. weeks duration. Computer courses: from introduction to PC to highly specialized courses. Sweden. Training replacement. TheQuasi-experimental. All the programs initially According to the author, a Training replacement. aim is to enhance skills of Matching. reduce participants' likely factor behind these Labor market programs employee while providing Differential program employment probability in thedisappointing results is the to face the deep and an unemployed individual effects and the effectshort term (lock-in-effect: lessuse of such types of sudden recession of the with work experience in a of joining a given search because occupied with programs simply as a way to early 1990s. regular job. program vs. waiting training). Positive findings on re-qualify for unemployment Period 1994-1999 longer in more long-term employment benefits. [See also Wage unemployment. prospects are confined to job subsidies] subsidies alone. [Sianesi, 2001] Sweden. Many programs: Hazard Regression Programs in which the Comparison of effectiveness To increase unemployedsubsidized on the job Model. Focus on participants obtain subsidized across eight programs. workers' employment training, trainee what type of work experience and training prospects augmenting replacement schemes, programs reduces provided by firms have better their human capital or work placement scheme, the duration of outcome than classroom work habits. labor market training by unemployment on vocational training. The 1995-1999 providing skills necessary participants, and, if relative efficiency is similar [Carling and for particular jobs in the relative across demographic and skill Richardson, 2001] excess demand. Some efficiency across groups and independent of the programs might be similar programs vary timing of placement. to the above ones. between Difficult to be precise. demographic and skill groups. United States. Programs are quite Non-experimental. Short-run employment Workforce diverse. Five of the impacts are positive for seven Development System. programs serve job-ready of the nine programs and Period 1997-2000 adults (unemployed): negative for the other two. Nine programs in community and technical Impact ranges from 3% to Washington State. job training, private career 12% of higher probabilities of [Hollenbeck, 2003] schools, apprenticeships, employment. JTPA Title III programs Short-run earnings impacts and community and are insignificant for four of technical college worker the programs, negative for retraining. Two of the two and positive for the programs serve adults remaining three. with employment barriers. The benefit cost analyses show positive results for all programs. 79 Training (LT Unemployed) Labor Market Problem Intervention Design Type of Evaluation Result Comments Transition Countries Intervention Design Type of Evaluation Result Comments Bulgaria. Professional training for Matching pairs and Both guaranteed or non- Professional training for unemployed in logistic regression guaranteed job training unemployed accordance with labor adjusted estimates toincrease employment Period 1998-1999. market need, with an analyze re- compared with the control (See Public Works) allowance paid equivalentemployment group, with a 10% of [Walsh, K. et. al., 2001] to 60% of the individual'sprobabilities of greater probability. benefit entitlement. It can individual be job guaranteed or non- participants. guaranteed. Macedonia. The target group for this Quasi-experimental..The probability of being "New Employment" training was unemployed Logistical regressionemployed was Period 1994-2000 people in general analysis. approximately 14 times [World Bank, 2002b] recruited through higher for participants in advertisements. the program compared to non-participants. Poland. Public financed further Matching Positive short-term effect Public financed further training and retraining (no on post-treatment training and retraining. information on selection). employment for both Period 1992-96 Relatively short courses males and female in [Kluve, et.al. 1999] comparison with non- participation. Only positive effect for females in medium term. Poland. Publicly financed off-the- Exact matching For both men and women Little information on the exact Publicly financed off-the-jobjob training and using previous labor there is a positive and nature of the program. Refers training and retraining. retraining. Training for market history. significant effect on to previous studies. Similar programs than the individuals that have employment probability Analysis seems to be very well ones above but using experienced at last one in comparison to the done, although the sample sizes subdivisions in matched spell of unemployment control group (13.8% are small. samples. during the observation greater probability). Period 1992-96 period. [Kluve, 2002] Poland. Training courses usually Two methods used: Training has positive The Polish economy was Training for unemployed. last from 3 to 6 months Matching and effects for both men and growing strongly throughout at Period 1990-1997 and most are taught in multinomial logit women, but the effect is least the latter part of the [Puhani, 1999] classrooms. Training duration models. only significant for men. period in analysis. within firms is possible asThe duration models These results are identical well. include unobserved for both models. heterogeneity. Romania. Eligible clients for this Quasi-experimental. Positive and significant Training and Retraining service could receive op Propensity Score impact on employment. Services. to nine months of Matching. The program increased Period 1999-2001 training. Cost of training the likelihood of [Benus and Rodriguez- was limited to $560 per employment by 10 %. Planas, 2002] unit. The program had no impact on earnings. The effect is entirely attributable to the large impact for females Slovakia. Publicly financed Multivariate Positive effect on Publicly financed retraining retraining for duration model with transition to employment for unemployed. unemployed. Average selection (to account for both men and women. Period 1992-1997 duration of courses is 8 for endogenous No information on [Lubyova, 1999] weeks. No information onprogram take-up). medium- to long-term content. effects. 80 Developing countries Intervention Design Type of Evaluation Result Comments Argentina. One random sample of the Randomized Training had no Participants in the temporary Proempleo experiment. program received a experiment. Use significant impact. workfare program were Heavy retrenchment by voucher that entitled a 2SLS to account for Voucher recipients had a divided into three groups: main employer in private sector employer to endogenous significantly higher Vouchers, Vouchers and "company towns". a wage subsidy that compliance (mainly probability of training, and control. 30% of Period 1998-2000 covered part of the total for the training employment (6.1%), those offered the training did [Galasso et. al., 2001] wages paid to the component). though their current not take it up. employee. incomes were not higher. A second sample received The impact was largely only training. confined to women and younger workers. Colombia. SENA is aimed to Quasi-experimental. For all youths and adult The study is focused on "Servicio Nacional de promote efficiency in the Matching. males SENA has no training activities but not Aprendizaje" (National labor market through job significant impact in the necessarily for unemployed Learning System) training ­training for short or long run. For workers. It also includes a (SENA). youths, labor adaptation adult females there are comparison of other public Period 1996-1997 for change and aid to positive but not programs (aggregated) and [Medina and Nunez, displaced population-, and significant impacts in the private job training programs. 2001] in a lesser degree job short-run and larger and search and advise to close to significant in the employers about trends in long run. No significant the labor market. impact on earnings. Mexico. Trainees receive minimumQuasi-experimental. The program does not This evaluation is a Retraining Program on wage during the training Hazard model to the have large positive effectsreevaluation of two previous Employment and Wagesperiod (6 months) and the length of on employment or ones in 1992 and 1994. The (Probecat). local employment office employment search earnings for participants. results of this third evaluation Targeting at increasing provides placement and probit model on The study even shows led to fairly different findings earnings and assistance. program negative impacts -but and policy conclusions in the employment for participation, using insignificant- reducing thesense that previous unemployed and program availability length of unemployment evaluations found positive displaced workers. at the local level as in the case of men for the and significant results of the Period 1984-1998 control variable. school-based training. It program on employment and [Wodon and Minowa, is not favorable in terms earnings. 1999] of cost-benefit. The program seems to be providing temporary safety nets through the minimum wage stipend rather than training. Mexico. Probecat also has two Quasi-experimental. Impact of program varies This study goes further Probecat. modalities. The first, Matching. across personal looking at impact on: See above. school-based training, characteristics, regions -Frequency and duration of Period 1984-1998 consists of formal courses and institutions. The on- subsequent unemployment [Calderon-Madrid and and training offered in the-job training is more and income profiles Trejo, 2001] institutions associated effective than the school- -Differentiation by with government or based training. geographical zones. private industry Institutions run by private organizations. The second, sector outperform mixed training, consists of government-run ones. on-the-job training in Hourly wage earnings for firms. Government pays women were on average an stipend and firms are 12% greater compared required to hire at least with non-participation. 70% of trainees. For men, earnings are lower for participants. Cost benefit analysis is positive (provided number of days of work is taken into account). 81 Retraining in Cases of Plant Closures & Mass Layoffs Labor Market Intervention Design Type of Evaluation Result Comments Problem OECD / Developed countries Australia. Generous labor Discrete-time event The length of training hadMany of those laid-off had no The Australian textile, adjustment arrangements history on panel large and significant formal qualifications and had clothing and footwear (not means tested) that data. Does not negative effects on the problems with English (TCF) sector diminishedencouraged retrenched account for selectionlikelihood of finding a joblanguage literacy. There rapidly due to easing of workers to participate in into training. for both men and women. appears to be evidence that trade restrictions with retraining. No information The same case for training did enhance the resulting large laid-offs. on type of training vocational training, prospects of those with Period 1989 - 1993 attended. although the effects were relatively good re- [Weller, 1999] not as significant. employment chances. Transition Countries Intervention Design Type of Evaluation Result Comments Bosnia & Herzegovina.The Emergency Quasi-experimental. The program increased Assist in the Demobilization and Included a survey to the likelihood of reintegration of Reintegration Project participants and non-employment by 43%. demobilized soldiers includes Labor Market participants and thenWage and salary and displaced workers Information Data Base, regression analysis increased by 45%. into the civilian Education and Retraining using dummies for The program had a workforce and to Services, Counseling and participation in the positive and statistically increase economic Job-Finding Services, and program. significant impact on all productivity. Management Assistance subgroups by gender, age Period 1996- 1999 for demobilized soldiers and level of education. [Benus, J., et. al., 2001] and displaced workers. Developing countries Intervention Design Type of Evaluation Result Comments China. Shenyang: Uniform one OLS (although Insignificant employment Xiagang are persons who are Reform of China's month duration in propensity-score andeffect in Shenyang, but laid off, but are still attached state-owned-enterprise classroom. Free training log-odds ratio improved the likelihood to employer to some degree (SOE) sector with given to all laid-off matching were also of employment in Wuhan.(housing, health insurance, resulting downsizing workers. Quality of used). No significant impact on etc). and rising labor program varied greatly. earnings in either city. redundancy. Wuhan: Variation in [Bidani, 2002 #3] length of training. 82 Training Geared Towards Youth Labor Market Problem Intervention Design Type of Evaluation Result Comments OECD / Developed countries Canada. YSC is designed to provide Quasi-experimental. Participation in YSC had no Per participant cost of Youth Service Canada unemployed and out-of-school Multivariate analysis. statistically significant effect the program is $8,277, (YSC). youths between 15 and 30 Linear regression on earnings. slightly higher than Period 1994-1997 years of age an opportunity ­ models. In the short term, participants other similar programs. [HRDC, 1999] within a project that lasts on spent less time in the labor average 6 to 9 months- to gain force and less time employed relevant work-related than would have been the case experince. in the absence of the program. Employment insurance benefits were $183 lower per year as a result of participation in YSC. France. No description of types of Dynamic multinomial Training schemes have no For people less than 25 High levels of youth training received. logit on a panel data effects on the relatively training is the main unemployment in France. set. Considering likelihood of getting a stable (almost only) way of Period 1990 - 1992 transitions between 6 job and do not increase the receiving benefits. For [Magnac, 2000] labor market states to duration of a job spell if a job them training schemes contrast short run is found. are substitutes of consequences in terms unemployment, of stable employment employment and of training spell from schooling. consequences of unemployment spell. France. Six different work/training Mixed proportional Participation in a private sectorIt is not always clear Youth unemployment. programs, which are hazard model with program increases likelihood which type of program Period 1986 - 88 aggregated into two categories, multiple spells and of leaving unemployment for there is referred to and [Bonnal, 1997] those provided by/in private states and unobserved those with low educational how much aggregation firms and those provided by heterogeneity. levels, while there is no effect is used in different the state. The latter is referred Evaluation focused on with higher levels. The public parts of the paper. The to as "workfare", but does durations of trainees' sector programs has no effect study looks only at include a training program. subsequent spells of on those with low educational males. unemployment and attainment and significantly employment. negative on those with higher levels of education. France. Programs are divided into Quasi-experimental. There are no significant Focuses on the relative Youth unemployment those in the private and the Uses ratio of propensitydifferences between the effectiveness of the This study examines the public sectors. The ones in the score to look at the different programs when different programs same programs that aboveprivate sector have a higher effects of multiple looking at entry into stable rather than comparing one. amount of vocational and treatments. Competing- employment. Among the with no participation Period 1986-1988 specific training than those in risks duration models programs the on-the-job (which is mentioned as [Brodaty, 2001] the public sector. The lengths are used to calculate training programs in the a topic for further of the different programs vary propensity scores. private sector seems to research). One of the significantly. perform better than the programs evaluated is programs in the public sector. closer to a public works program than a training program. Sweden. Two main programs: Quasi-experimental. Both youth practice and labor Results for youth During the crisis in the a) Youth Practice: directed atMatching, OLS, Probit. market training have negative practice are explained 1990s, youth unemployed youth aged 18-24. short-term effects (one year by insufficient planning unemployment grew It was a subsidized program after the program starts) on and follow-up, as well rapidly. and participants were placed in earnings and employment. The as low-qualified tasks Period 1991-1997 both private and public sectors. long term effects are mainly that did not provide any [Larsson, 2000] b) Labor market training for zero or slightly positive. human capital people under 20: to improve The results also suggest that accumulation. skills of unemployed job workplace practice is more seekers. effective that classroom training. 83 Training Geared Towards Youth Labor Market Problem Intervention Design Type of Evaluation Result Comments United States. The Job Corps program Experimental. Job Corps generated positive Job Corps has a large The Job Corps program. operates under provisions Data from periodic and statistically significant residential component; about 118 Job Corps of the Workforce interviews earnings impacts beginning in 88% of the students live at campuses located Investment Act of 1998. conducted over a the third follow-up year, which the centers while enrolled. throughout the United Participants are between four-year follow up persisted through the end of theThe program has a high States. 16-24 most without high period. Impacts for four-year period. Average degree of federal direction Period 1994-2000 school diploma. Services youth characteristics weekly earnings for with many regulations and [Gritz and Johnson, are delivered through Job were estimated by participants were $16 higher contractual oversight. 2001] Corp centers that provide: comparing the than control group members vocational training, average outcomes of (12% earnings gain). The academic instruction, program participants program is cost effective. residential, health and other and control group services. The average members. participation period is 8 months. United States. Replication in 12 sites Quasi-experimental. The CET model had few or One explanation for the Center for Employment across US of a CET that 30 month follow-up negative effects on difference of results by Training (CET) started in San Jose, Puerto survey. employment, earnings and gender is that the majority of Replication Sites. Rico. CET enrolls trainees income during the first 30 young women participated Period 1997-2002 with little prescreening, months. However, CET in in accounting and office [Miller, et.al. 2003] providing training in a some sites led to an increase in training. In contrast, young worklike setting, and employment and earnings for men received training in requiring a full-time females but had the opposite nonclerical jobs. commitment from trainees. effect for males. In the sites that were less successful in implementing the CET model, the program reduced employment and earnings for less educated youth and for those younger than age 19. Developing countries Argentina. The program offers training Quasi-experimental. Program impacts on earnings Weakness of the program: "Programa Joven" to facilitate formal labor Propensity score were statistically significant forNo incentive for funding Increasing force participation. The Matching estimator. young males and adult females.agency to focus on quality employability and program covers two stages: Impact on employment was of training (similar to productivity targeting technical knowledge phase statistically significant for adultChilean and Peruvian activities to youth aged and internship phase. To females only. experience). 16 to 29 in the lower carry out training the The cost-benefit analysis socioeconomic levels. "Ministerio del Trabajo" suggests that at least 9 years of Period 1996-1997 hires educational duration of the earnings impact [Aedo and Nunez, institutions through an for the program are required to 2001] international bidding have a positive net present process. value for the groups with statistically significant results. Brazil. The program focuses on No information The impact on unemployment Evaluation results presented National Plan of marginalized people, but provided. and income is quite small. In in a World Bank publication Professional Education not necessarily youths It is some areas the program had (not from the primary (PLANFOR). financed through a U$S 20 positive and statistically source). To provide training for billion unemployment fund. significant impact on The study was carried out marginalized youths. Around $300 million of that unemployment but not on only in the metropolitan Period 1996 ­ 1998 goes to training incomes of those already regions of Rio de Janeiro [Paez de Barros, 2002] approximately 1.3 million employed. The new jobs for and Fortaleza. workers. The plan works participants need to last more with a fund overseen by a than 17 months for the program tripartite committee of to have a net positive benefit. government, enterprises. 84 Training Geared Towards Youth Labor Market Intervention Design Type of Evaluation Result Comments Problem Chile. The basic service consists Experimental. Positive and significant "Chile Joven" of a training and Surveys to treatment effects increasing To provide training and occupational practice and control group. employment among skills development package (6 months) aimed Logistic Regression youths. The subprogram among unemployed and at attaining a semi-skilled Models. of alternating (dual) underemployed low- level in specific trades. training ­that rotates income youths between The labor practice training at a technical 15 and 24 years. component is carried out school and at a firm, with Period 1995-1997 in a firm that assumes a a labor contract- presents [Santiago Consultores tutoring role under the the largest effect. Impact Asociados, 1998] supervision of the is higher among youths executing unit (in central aged 20 to 24 -- most government). likely for their higher education level-- and among men. Peru. Based on "Chile Joven" Quasi-experimental. Positive and significant The study included "Projoven" model. "ProJoven" is Matching and net effects. The program participants that had Supplying semi-skilled addressed at young people difference-in- allows an additional 6% completed the training six training and labor between 16-25 years from differences. of total participants to be months earlier. experience to low- low-income households, employed. As opposed to Use of Duncan Index to income young people in with little or no labor non-treatment The estimate occupational specific trades in experience, unemployed program also increased segregation by gender. demand in the or underemployed. In hourly wages in by18%. productive sector. particular, ProJoven Participants work 5.5% Period 1996-2000 focuses on marginalized additional hours per [Grade, 2001] young urban populations week. The program also that find the greatest reduces occupational difficulties in getting a segregation by gender and job. increased the probability of working in larger firms. Uruguay. Short-term training and in-Experimental. Probability of Not mentioned how long ago Uruguay Pro Joven. firm subsidized Surveys to treatment employment increased by ex-post group had completed Period 1996-1997 employment for 6-12 and control group. 60%. Reduction of search the program. The study [Naranjo, 2002] months. Young population time by 8.5 months, with summarizes evaluation [Fawcett, 2002] aged 17-24. strong gender impact. results. Primary sources have Quality of new not been found. employment is higher. Increase in salary depends on type of training but varied between 20% and 18% of higher earnings with the control group. 85 Wage / Employment Subsidies Labor Market Intervention Design Type of Evaluation Result Comments Problem OECD / Developed countries Australia. Provided support for Logit on the probability For both men and women The period examined is not Period 1993- 1996 long-term unemployedof staying in the labor there is a significant and discussed, so it is difficult to [Stromback et al., 1999] through wage force and on positive effect on the establish how long after subsidies paid to employment. probability of working. participation that status is employers. The level observed. and duration varied with the jobseeker's level of disadvantage. Belgium. Pure wage subsidies, Hazard model of length The impact of wage Not possible to disentangle [Cockx et al.,1998] which are often of employment with subsidies on job tenure is the subsidized period from the targeted at specific corrections for selection non-significant. after subsidy period. disadvantaged groups. and bias from sampling Maximum duration of method. payment is between 12 and 24 months. Canada. Self-Sufficient Project Experimental. Using a SSP increased full-time Project is focused on long- Finding policies that (SSP) offered a survey completed by employment and term welfare recipients. encourage welfare temporary earnings participants about three earnings. To receive the recipients to work New supplement to selected years after they entered earnings supplement, Brunswick and lower single-parent, long- the study. people had to begin British Columbia. term income assistance working full time during Period Nov. 1992- recipients. The the first year. By the Dec.1999 supplement was paid second year, the program [Michalopoulos, et. al., on top of earnings had doubled full-time 2002] from employment for employment. Most up to three continuous employment resulting was years, as long as the stable. SSP increased person continued to average earnings by about work full time and 30% over a three-year remained off income period. In the fourth year, assistance. the program increased earnings by $52 per person each month. Canada. Self-Sufficient Project Non-experimental Equilibrium wages fall The study evaluates and Finding policies that (SSP). (See above) partial and general for those treated by the compares the predicted partial encourage welfare equilibrium estimates. program after the and general equilibrium recipients to work but introduction of the effects of SSP to those in the preserve and adequate earnings supplement, as SSP experiment. (See above). safety net. workers are willing to New Brunswick and the accept lower starting lower British Columbia. wages. General Period Nov. 1992- equilibrium results, in Dec.1999 contrast with partial [Lise, Seitz and Smith, equilibrium, show an 2003] important dead weight loss of the program. In New Brunswick is approximately only 10% of the size of the net gain predicted by the partial equilibrium analysis. 86 Wage / Employment Subsidies Labor Market Intervention Design Type of Evaluation Result Comments Problem Canada. Insurance against extreme Random assignment Small positive significant effect Virtually no effect on the Earnings earnings losses if another experiment. Small- on rapid re-employment (the offeramount or duration of Supplement Project job found quickly and there scale test; 5912 lasted for six months). The unemployment benefits (ESP). is a reduction in earnings. person in total (half program increased the percentage received by the ESP group Attempts to help Supplement payments up to in control and half inof displaced workers who becamemembers. 20.5% of those hit by 75% of difference (max treatment). re-employed full time during this treatment group received economic $250 per week). Supplement period by 4.4. percentage points. supplement payments. adjustment. paid to the unemployed. This reflected a shift from part- Period 1995-1996 time to full-time employment and [Bloom, 1999] an increase in overall employment. Average earnings during the 15 months follow-up period were $682 or 4.6% lower than they would have been otherwise. Germany. Public Employment Quasi-experimental PEP participation reduces future The PEP employment might Large increase in Program (PEP). The Matching. unemployment. be used as a "trial period" to unemployment in objective is to create For men the reduction in determine the work East Germany after additional jobs by providing unemployment is mainly due to capacities of the participant. unification. a wage subsidy to an an increase in employment. For Period 1990-1994 employer targeted at women, the increase in non- [Eichler and previously unemployed or at employment (out of labor force) Lechner, 2000] people who might become is substantial as well. unemployed. The subsidy can be up to 100% Great Britain. After 6 months of Quasi-experimental. Young unemployed men are The New Deal treatment New Deal for Youngunemployment, 18-24 years Difference on about 20% more likely per period effect comprises the effects People. olds are mandated to enter a differences. to gain jobs as a result of the New of both the job Period 1998- 2002 "Gateway" period where Main focus on Deal (that includes wage subsidy assistance/monitoring [Van Reenen, 2003] they are given extensive job evaluating the and job search assistance). element of New Deal and search assistance. If they are success of the The wage subsidy appears to havethe wage subsidy element. unable to obtain program in moving had a significant impact on unsubsidized job, then they people into jobs. For increasing jobs, at least in the first can enter one of four New data restrictions, the few months on the program. Deal options: job subsidy, evaluation only The cost benefit analysis suggests training, environmental task considers male that the program is worth force or voluntary work. participants. continuing. Switzerland. Two programs: A non-profit Quasi-experimental. Subsidies for temporary jobs in Period 1996-1999 employment program (EP) - Matching with competitive firms (TEMP) have [Gerfin, M., M. subsidized job in the non multinomial probit higher impacts in comparison to Lechner, et. al., profit sector- and a subsidy model. the "classical" employment 2002] for temporary jobs (TEMP) programs (EP) operating in areas in firms operating in without much private competitive markets. competition. A year and a half after the start of the program, TEMP participants had 9% higher employment compared to EP or non- participants. Sweden. Employment subsidy Quasi-experimental. Compared to waiting longer in The study evaluates the Labor market eligibility for anyone older Propensity score open unemployment, all the differential performance of programs to face the than 20 and with 6 or more multiple treatment programs reduce participants' the six main types of deep and sudden months unemployed. matching. employment probability in the Swedish programs that were recession of the For private sector employers short term. Positive findings on available to adult early 1990s. only. Employer incentives: more long-term employment unemployed workers Period 1994-1999 grant 50% of labor cost up prospects. entitled to unemployment [Sianesi, 2001] to fixed amount. benefits in the 1990s. 87 Wage / Employment Subsidies Labor Market Problem Intervention Design Type of Result Comments Evaluation United States. After 1996 welfare reform, a Quasi- Participants had higher earnings The study summarizes Welfare-to work number of states replaced experimental. and lower welfare payments than results of 20 welfare-to- programs. voluntary welfare-to-work non-participants, but generally work programs across [Michalopoulos, at. Al. programs that emphasized had the same combined income eight states. Considered 2001] education and training with from earnings, welfare, and Food as wage/employment mandatory programs that Stamps. As a result, the programs subsidy program stressed quick employment. generally neither increased nor because there is also a decreased combined income. The subsidy involved but programs increased earnings paid directly to about as much for the more individuals as an disadvantaged groups as for the incentive to work. less disadvantaged groups. United States. Bonuses might be offered Statistical model Targeting bonus offers with Cost-effectiveness Reemploying bonus to using worker profiling in a to estimate the profiling models similar to those analysis of unemployment insurance two-step process. First, probability of in state Worker Profiling and reemployment bonuses. claimants. permanently separated workers benefit Reemployment Services systems Pennsylvania and are identified by screening out exhaustion. can improve their cost Washington States. those expecting recall to their effectiveness. The best option is a Period 1988-89 previous employer and those low bonus amount with a long [O'Leary, et. al., 2003] subject to an exclusive hiring qualification period, targeted to hall agreement. The second the half of profiled claimants step is to determine which of most likely to exhaust their UI these permanently separated benefit entitlement. workers are expected to be long-term benefit recipients. Transition Countries Intervention Design Type of Result Comments Evaluation Poland. "Intervention Works" is a Quasi- Subsidy has no effect on According to the author "Intervention works". program that in essence gives experimental. unemployment of females and a an explanation for these Rapidly increasing job or wage subsidies in the Exact covariate strong negative effect for males, results is that unemployment during amount of unemployment matching. being worse than non- participating in transition in Poland. benefit. These wage subsidies participation. intervention works or Period 1992-1996 are given to firms if they hire public works might [Kluve, 1999] an unemployed person. carry a stigma. Employers will not hire such persons as they perceive them as low productivity workers. A competing second explanation is "benefit churning". Poland. Wage subsidy (See above). Quasi- Subsidy has no effect on After initial contraction in experimental. unemployment of females and a the Polish economy there Exact matching strong negative effect for males, was a robust expansion. using previous being worse than non- Similar programs than the labor market participation. ones above but using history. subdivisions in matched samples. Period 1992-1996 [Kluve, 2002] Poland. Any type of job can receive Two methods There are significant negative Receiving wage During transition, Poland subsidy, which is paid out for 6used: Matching effects (it would be preferable subsidies also made experienced very high to 12 months. The completion and duration non-participation) of the program men less likely to leave unemployment in the of an "intervention works" job models. The for both men and women, the labor force. beginning and thereafter lead to a complete renewal of duration models although the effect for men is an improvement. the 12 months unemployment include larger than for women. Both Period 1990 - 1997 benefits. unobserved models support these results. [Puhani, 1999] heterogeneity. 88 Wage / Employment Subsidies Labor Market Intervention Design Type of Evaluation Result Comments Problem Slovakia. Socially Purpose Jobs Multivariate duration Negative effect on transition Socially Purpose Jobs. are subsidized jobs in model with selection to employment for both men High share of long-term the private sector. (to account for and women, although the unemployed and Main forms of support endogenous program effect for men is not relatively low outflow are subsidies, interest take-up). significant. to jobs. repayments and loans. Period 1992-1997 [Lubyova, 1999] Developing countries Intervention Design Type of Evaluation Result Comments Argentina. "Proempleo" Randomized Significant effect on Participants in the temporary "Proempleo" experiment. Received experiment. Use 2SLS employment in the private workfare program were experiment. a voucher, which to account for sector (six per cent relative divided into three groups: Heavy retrenchment by entitled a hiring endogenous to non-participation), but no Vouchers, Vouchers and main employer in employer to a wage compliance (mainly forsignificant effect on other training, and control. A "company towns" in subsidy of $150/monththe training outcomes. This effect possible reason for the low Argentina. for those 45 and above component). disappeared in the third take-up is the costs associated [Galasso, et. al. 2001] and $100/month for survey round (after 12 with registering an employee. those younger than 45 months). Women and those If the "letter-of-introduction" (min. wage was $200). under 30 were the ones who effect is important the results saw significant impacts. are unlikely to hold when Proempleo experiment was scaling the program. cost effective, given that take-up of the subsidy by employers was low.. Effect thought to be due to a "letter-of-introduction" effect. 89 Public Works Programs Labor Market Issue Intervention Design Type of Result Comments Evaluation OECD / Developed countries Belgium. In Belgium welfare agencies Duration analysis The study proposed a grouping Progression from welfare is not The Social are subsidized to employ using grouping/ estimator of the SE effect that necessarily to employment, but Employment (SE) welfare recipients for a aggregation to eliminates selection bias. The may also be to unemployment or program. Welfare period sufficiently long to account for estimator is consistent, even if theother states. agencies provides both entitle them to benefits of selection. This selection into SE depends on the welfare and the contributory social paper investigates average unobserved employment programs. insurance program. This the effect of SE characteristics of welfare [Cockx, 2001] type of employment is calledon the exit rate recipients in a region and in a Social Employment (SE). from welfare. welfare duration interval. The SE program provides participants with minimum wage jobs, mostly in community services. Canada. Jobs that are of a short-term Focuses on the -CDJCP were generally found to Evaluation lessons from a Canadian direct job nature to reduce the number cost of job be appropriate tools ­in sense of recompilation from programs creation programs of unemployed and to creation as an helping employment and implemented between 1977-1996. (CDJCP). reduce dependence on social evaluation earnings- for certain The study presents costs of the Period 1977- 1996 programs. measure and disadvantaged labor force groups: programs over time. Incremental [HRDC, 1998] -Canada Works results and poor population and women. cost per job varies from $4,200 to -Local Initiatives Program. conclusions from $26,700 across programs (1986 previous constant dollars). evaluations. Cost effectiveness of CDJCP improved over time due to developing better program design, dropping ineffective elements and expanding better elements. Transition Countries Intervention Design Type of Result Comments Evaluation Bulgaria. Temporary Employment Matching pairs The involvement in the program Among other ALMP in Bulgaria, Temporary Program provides temporaryand logistic hardly improves this has the lowest net impact. In Employment Program. work for a maximum of 5 regression (2.5% net impact, statistically addition, the costs per additional Period 1998-1999 months in public or private adjusted estimatessignificant) the chance of the job are the highest of all programs [Walsh, K. et. al., sector projects. to analyze re- unemployed to have a regular job.in Bulgaria. 2001] employment This program is particularly probabilities of effective for disadvantaged individual workers, the older unemployed participants. and the long-term unemployed. It is also effective in regions of high unemployment compared to control group. Macedonia. The basic aim was to carry Quasi- A net employment impact of 34% Public Works out some public works and experimental.. (the difference between Period 1994-2000 to give occupation to the Logistical participants and non-participants [World Bank, 2002b] unemployed. regression who currently have a job). analysis. Poland. Direct job creation through Matching The program has little effect on Poland spends little on ALMP The study evaluates public works mainly, but not employment for both males and and has low inflow into ALMP. three main ALMP: exclusively targeted at long- females. For men public works Wage subsidies and public works training, "intervention term unemployed. Jobs have and "intervention works" have received the highest funding. works" (kind of job very low skill requirements. significant negative treatment According to the authors, while subsidies) and public Recipient qualifies for effects, while participation in stigmatization might have some works. additional unemployment "intervention works" does not role to play ­employers will not Period 1992-1996 benefit from staying in a affect women's employment hire such persons, as they [Kluve, 1999] subsidized job for 6 months. probabilities. perceive them as low productivity workers-, "benefit churning" explains most of the negative effects of employment programs. 90 Public Works Programs Labor Market Issue Intervention Design Type of Evaluation Result Comments Poland. Typical jobs are in Two methods used: For both men and women The effects for women are Direct job creation construction or cleaning. Matching and there is a significant based on a very small sample. targeted on the long-term Duration is between 6 and multinomial logit negative effect on the unemployed. 12 months. Recipients duration models. probability of exiting Period 1990 - 1997 qualify for additional The duration models unemployment, although [Puhani, 1999] unemployment benefit from include unobserved the effect is larger for staying in a subsidized job heterogeneity. men than for women. for 6 months. Romania. Public works projects had a Quasi-experimental. No impact on wage and Public Works Communitymaximum cost of $50,0000 Propensity Score salary employment or on Job Creation Program. and covered the cost of Matching. self-employment. Modest Period 1999-2001 supervisory personnel and reduction in the length of [Benus and Rodriguez- up to 6 months of program current unemployment Planas, 2002] participants' stipends. spell. Insignificant impact for both males and females. Slovakia. Public works program aimedMultivariate Positive effect on the Publicly Useful Jobs at less qualified workers. duration model. transition to employment (PUJ) are short-term jobs The program covers wage for both men and women. in public works costs and social insurance No information on (community works). contributions. Maximum medium- and long-term Period: 1992-1997 duration of jobs used to be 6 effects. [Lubyova, 1999] months, but now is 12 months Slovenia. Direct job creation for the Matching two-stage Immediate positive High unemployment unemployed under the model with impact on transition to resulting from transition. auspices of a public or non- identifying employment, i.e. [Vodopivec, 1999] profit organization. Jobs not variables. immediately after public to exceed one year (not works job has ended. The binding for disabled or over effect becomes 50 years of age). insignificant if no job is found after 3 months after program. Ukraine. Public works are used for Matching with Cobb Public works has a larger The study evaluates impact of During transition, there disadvantaged groups at the Douglas positive effect on the the total measure of labor was an overall labor market, in particular specification. hiring of the unemployed market policies in Ukraine but restructuring of economy long term, and low-skilled than training just after the presents the results of public with new conditions of unemployed. completion of the works program in contrast with the reallocation of jobs program. Public works training and retraining and workers between a program by itself is programs, public employment declining public sector effective (significant) services, and others. and emerging private when participants find a sector, and non-matching match with available of this process. vacancies just after the Period 1995-1999 completion of the [Olga, 2000] program. Developing countries Intervention Design Type of Evaluation Result Comments Argentina. Trabajar aims to provide Propensity-score The average net gain in Losses to those who left the "A Trabajar" program. short-term jobs at relatively matching. income is about half of program were sizable (about ¾ Sharp increase in low wages. This is done the average Trabajar of the gross wage of the unemployment in the through socially useful wage. Percentage net gainprogram) within the first six mid-1990s in Argentina. projects in poor areas; most for the poorest 5% is months, but decreasing after Period 1990-1997 directed at repairing and 74%. Average gains are that. [Jalan and Ravallion, developing the local very similar between men 2002] infrastructure. Maximum and women, but are duration is six months. higher for younger workers. 91 Micro-Enterprise Development Labor Market Intervention Design Type of Evaluation Result Comments Problem OECD / Developed countries Canada. SEA is a program, Non-experimental. Short-term business survival An important concern Self-Employment running up to 52 weeks, Survey instruments rates are high. In the year uncovered by the evaluation Assistant (SEA) of self-employment and comparison following program is that about half of the Program designed to assistance with financial group. participation, participants clients are subsidized to start promote self-sufficiencysupport for people who experience positive earnings a business they would have in the labor market are on, or who have been effects ($213 more on weekly started anyway through self- on Employment earnings compared to control (deadweight). employment. Insurance. group), though for some [Graves and Gauthier, participants the new business 2000] is not a sufficient source of income. Participants increased by 13 hours per week between the pre and post-program period. Germany. "Bridging allowances" Simultaneous Subsidized businesses have a Not clear whether Persistent high for unemployed proposingmodeling of survivallower survival probability identifying variables are unemployment in new business for a period of company to one than non-subsidized used. This study investigates Germany. Significant of 26 weeks at the rate of year and the businesses with similar firm survival and easing of conditions for the last paid employment growth characteristics. There are no employment growth of start- receiving support for unemployment assistance. rate of surviving effects of subsidies on ups from unemployed people setting up a business in No loss of right to companies. employment growth for in East and West Germany 1994. unemployment payments surviving companies. as promoted by the Work [Pfeiffer and Reize, or assistance by a Support Act. 2000] transition into self- employment and the right to unemployment assistance is extended by 26 weeks. Sweden. Approved business plan Logistic regression Self- program is associated Increase in gets grant for six months model and hazard with much less unemployment in equivalent to 80% of pre- regression model to unemployment in future. The Sweden in the early unemployment earnings. compare the risk of re-unemployment is 1990s. There was also Subsidized employment efficiency of the more than twice as high for an increase in program is also for the self-employment the subsidized program unemployment for same amount. Both program with the compared with the self- unskilled workers. programs have the same traditional employed program. The large Period 1995- 1999 duration. subsidized positive effect is limited to [Carling and Gustafson, employment workers of Swedish origin. 1999] program. United States. Self-Employment Linear regression Participants in Maine, New Self-employment Assistance (SEA) analysis. The Jersey and New York were 19 opportunities to Programs. States assist objective of the times more likely to have unemployment unemployment insurance study is to compare been self-employed at any insurance claimants. claimants in establishing longer-term time post-unemployment as Period 1995- 1999 themselves in self- outcomes compared to non-participants. States: Delaware, employment. States experienced by For participants 25-36 months Maine, Maryland, New provide entrepreneurial participants with from initial program Jersey, New York, training, business non-participants. enrollment, 58% (NY) and Oregon and counseling and technical 60% (NJ) were either self- Pennsylvania. assistance to SEA employed and/or wage /salary [Kosanovich, et. al, participants. employed. 2001] The cost of program services ranges from $200-2,000 per participant. 92 Micro-Enterprise Development Labor Market Intervention Design Type of Evaluation Result Comments Problem Transition Countries Intervention Design Type of Evaluation Result Comments Bulgaria. Program provides a lump Matching pairs and This program had the The period between Period 1998-1999. sum equal to the logistic regression highest net impact among participating in the self- (See Training) remaining part of adjusted estimates toALMP in public sector, inemployment program and the [Walsh, K. et. al., 2001] unemployment benefit, analyze re- contrast with non- evaluation is short. The which unemployed person employment participants, suggesting author suggests re-evaluating receive after approval of probabilities of that the program was the program one year later. business plan. Also offers individual delivering sustainable business skills training, participants. businesses or jobs. It etc. increases the probability of re-employment by 43%, compared to control group. Romania. Displaced workers who Quasi-experimental. Positive and significant Small Business start or operate a small Propensity Score impact on employment Consultancy and business are eligible to Matching. (6%) and on earnings. Assistance Program. receive legal, marketing, This result is entirely Period 1999-2001 sales, financial and attributable to the large [Benus and Rodriguez- consulting services. impact for females Planas, 2002] 93 Melvina E. Clarke N:\MELVINA\Betcherman, Olivas, Dar-Impact Evaluation.Final-1.doc January 15, 2004 4:31 PM 94