CODING BOOTCAMPS: GUIDE FOR PRACTITIONERS 2 CONTENTS INTRODUCTION 1 OBJECTIVE OF THIS TOOLKIT 1 WORLD BANK’S DECODING BOOTCAMPS INITIATIVE 2 WHAT IS A CODING BOOTCAMP? 3 HOW TO ENABLE CODING BOOTCAMPS 5 STEP BY STEP 9 BOOTCAMP DESIGN 9 NEEDS ASSESSMENT 9 SELECTION OF TARGET BENEFICIARY GROUP AND OUTREACH STRATEGY 11 12 SELECTION OF OPTIMAL MODEL AND PROVIDER 13 VENUE FOR A BOOTCAMP 14 PARTNERSHIPS WITH LOCAL ECOSYSTEM PLAYERS 16 BUSINESS MODELS 17 IMPLEMENTATION 20 LEARNING PROCESS 20 CURRICULUM DESIGN 21 COMMON CHALLENGES 24 POSTBOOTCAMP ACTIVITIES 26 GRADUATION CEREMONY 26 PLACEMENT OF GRADUATES AND OTHER TYPES OF POSTBOOTCAMP SUPPORT 26 EVALUATION AND DISSEMINATION OF RESULTS 28 CONCLUSIONS 30 NOTES 31 BIBLIOGRAPHY 32 ACKOWLEDGEMENTS 33 i ii INTRODUCTION OBJECTIVE OF THIS TOOLKIT This toolkit aims to help policy makers and private sector actors to effectively catalyze and leverage coding bootcamps for more and better employment and continuous learning in emerging markets. It draws on the practical experiences of selected bootcamp providers, engaged through the Decoding Bootcamps initiative, based in Colombia, Kenya, and Lebanon. It outlines the resources needed, in terms of human capacity, financial, organizational, and communication resources, to plan, implement, monitor, and evaluate a coding bootcamp in an emerging market, as well as ensure its sustainability and ownership within a local technology innovation ecosystem. This toolkit provides a step by step guide for policymakers and practitioners to develop coding bootcamp programs. The toolkit provides advice on what it takes to organize training from “scratch,” and hence help form an opinion on the ultimate model on which to settle. The toolkit is preceded by a section providing examples on how to create and promote an enabling environment for bootcamps. Each section ends with pertinent actionable guidance, listed in order of priority. While the toolkit is primarily oriented towards policy makers, it also serves to inform a broader set of stakeholders, including private and nonprofit organizations, international donors, and entrepreneurs. Moringa School 1 WORLD BANK’S DECODING BOOTCAMPS INITIATIVE Funded under the Jobs Umbrella Multi-Donor Trust Fund1, the World Bank launched an initiative to assess the key success factors for rapid technology skills training programs (in particular, coding bootcamps) and measure their impact on young people in terms of employment and employability. This initiative is specifically focused on exploring avenues to support upskilling, and thereby reducing unemployment problem in urban settings with traditionally large and growing young populations in developing countries. The initiative aims to test whether coding bootcamps can create employment for low-entry tech skills and reduce the skills mismatch in emerging economies. It has been piloted in three cities: Beirut, Lebanon; Medellín, Colombia; and Nairobi, Kenya. All three cities were selected because of the presence of vibrant local tech innovation ecosystems, relevant size of the low-income youth population (which helps extrapolate findings to other cities and even countries), and high youth unemployment. The impact of these bootcamps was measured through a randomized controlled trial (RCT) in Medellín, and qualitative studies (surveys and focus group discussions) in Beirut and Nairobi, carried out through collaboration with research partners from the local tech innovation ecosystems. The initiative has four main components: 1. An assessment of the impact of coding bootcamps on local, young jobseekers to improve employment and income generation opportunities in the short term; 2. A comparison of employment patterns between bootcamp participants and those in a control group who have not benefited from training; 3. Identification of key success factors of coding bootcamps and the preparation of a toolkit for implementing a bootcamp, based on an overview of existing tools and best-practice methods; 4. Guidance for policy makers in emerging markets on how to support the establishment, implementation, and growth of demand-driven rapid tech skills training to combat youth unemployment and boost the technology talent supply to meet private sector demand. Through these components, the initiative seeks to lay the foundation for a swift response to boost demand-driven labor market training that is necessary to tackle youth unemployment in today’s fast- changing world. Its ultimate goal is to establish best practice for rapid tech skills programs (coding bootcamps) that result in improved employability and responsiveness to private sector needs. The contents of the program and its results are available at: http://www.decodingbootcamps.org. More information on coding bootcamps’ origin, categories, principles and challenges can also be found in the World Bank’s first publication for the Decoding Bootcamps initiative (Mulas and others 2017). 2 WHAT IS A CODING BOOTCAMP? Coding bootcamps are intensive, short-term training programs designed to equip training participants with employment-ready programming skills for entry-level tech positions (Meng 2013)2. In essence, coding bootcamps combine characteristics of traditional vocational training programs with the intensity of military bootcamps for new recruits, intermingling socioemotional and tech skills learning in an intense manner, serving as “skills accelerators.” Coding bootcamps follow a structured process with three main characteristic features: 1) Intense rapid-skills training, 2) Experiential learning approach, and 3) Curricula based on, and continuously adapting to, industry’s demand. The coding bootcamp model is summarized in Figure 1. Figure 1: Coding Bootcamp Model A B C Selection Core Curriculum From Talent 3-6 months Pool Industry <-> Start-ups ENTRY-LEVEL EXPERIENTIAL LEARNING CONSTANT CONNECTION REQUIREMENTS TO DEMAND • Learning curriculum is experiential: • Minimum literacy and learning by doing • Curriculum is developed basic digital skills • Focus on soft 21st Century based on skills demand • Online prelearning skills: ability to learn how to learn, assessment of industry; prepares students for teamwork, problem solving, time- can be tailormade; Job application management, receiving and giving placement department feedback, presentation skills, self- constant dialogue with • Higher selection assessment, etc. industry criteria is attitude and willingness (“hunger”) • Short-term, intense training with • Guest trainers from industry to learn a cohort structure and heavy on ensure constant interaction simulations-based learning through learning experience • Some providers employ psychometric • Demo-day (pitch) to employers at • Agreements of regular test. end of course supply of employees and ad hoc recruiting • Data tools assess skill • Technical curriculum adapted levels and pretraining based on demand. New technical • Data tools provide visibility needs knowledge piece can be plugged of students progress and into experiential core methodology. graduates performance and support forecast demand/ • Data tools track student progress curriculum needs and match with employers Source: Authors. 3 A. Coding bootcamps are intensive and rapid skills training programs that typically last no more than six months. The majority of programs have a multistage application process and require admitted students to show full-time commitment to learning (in some cases students need to dedicate no less than ten hours per day, six days a week). The application process (A) is similar across bootcamps, relying on online applications and interviews. At first, interested applicants are invited to fill out a short questionnaire providing basic personal information on education, interests, employment, and so on. Then, eligible applicants are invited for an interview, either face-to-face or via telephone or online, to gauge their motivation to participate in the program. Some bootcamps are more selective than others, for example, previous coding experience might be required. B. Coding bootcamps’ teaching method follows a project-based, experiential learning approach. Students learn coding “by doing,” hence basic skills, such as teamwork, problem solving, and critical thinking, are integrated into the education process. Teaching sessions are usually combined with inspirational talks from industry specialists as well as mentorship, aimed at helping students to determine their career path. Unlike computer science courses offered by formal, tertiary educational institutions, coding bootcamps focus much less on teaching theory than on the practical application of concepts through the replication of a software engineering business environment. The latter is usually carried out through interactive teamwork or peer learning. Certain gamification aspects – from point scoring to competition with others – may be incorporated into the learning process to drive engagement among students, boost their spirit of competition, and master basic coding skills. The teaching philosophy of most bootcamps draws on agile software development methodologies, a set of principles that encourages collaboration, iteration, and self-organization within product teams. Bootcamp providers also emphasize that they are building “life skills,” enabling their graduates to be competitive irrespective of the industry in which they choose to work, for example, the ability to master new knowledge quickly and efficiently, effectively work in a team, meet tight deadlines, develop a growth mindset, and so on. Evidently, these “life skills” belong to the subset of future-proof socioemotional skills. Besides a strong push towards developing the socioemotional skills of students, bootcamps also embed career advisory services, so as to provide them with the right tools to find a job after training. C. Coding bootcamps’ curricula reflect current industry needs, with teaching subjects adapted to local demand. Coding bootcamps tend to pursue a “glocalization” approach in their curricula design: they create “in-house” study programs by mixing internationally recognized tech education products, for example, combining proprietary and free online courses, video tutorials, massive open online courses (MOOCs) with curricula that respond to the needs of the local tech ecosystem and reflect local cultural characteristics. Many programs are so connected to the local startup ecosystem that their trainers are at the same time employees within the industry. Coding bootcamps act as skills demand aggregators, crowdsourcing demand from multiple companies, ranging from small private firms to multinational corporations. It is common for bootcamp providers to base their assumptions of which Information Technology (IT) subjects or programming languages are in demand by monitoring publicly available online sources (for example, data from national statistical and labor agencies) and private market intelligence data. Typically, bootcamp providers engage local tech sector stakeholders – and even IT outsourcing companies located abroad – through meetings and surveys to understand existing and projected demand for certain skills. The frequency of curriculum reviews and adaptations vary, but it is much higher than in traditional curricula in academia. 4 HOW TO ENABLE CODING BOOTCAMPS Coding bootcamps provide an opportunity to disrupt the paradigm of traditional technical skills training.Technical training, in particular technology training, faces the challenge of continuously shifting demand because of rapid changes in technology. The bootcamp approach has shown potential to boost skill development, including quick results in terms of equipping graduates to compete for better jobs and building future capabilities to increase competitiveness in the labor market. However, the private sector alone is unlikely to tackle structural unemployment issues and the insufficient supply of in-demand digital skills over the long term. Evidently the formal education system has a role to play, and so do other tech ecosystem players, such as the government both at national and local level, IT companies, industry associations, innovation hubs, civil society, and others. As tech startup ecosystems tend to emerge and concentrate in cities, policy makers at city level are particularly well positioned to influence the supply of technology talent to fuel businesses that leverage technology. So how can policy makers build on the successes of bootcamps and further catalyze such skills development for greater, longer-term impact? The following recommendations provide a menu of policy options that aim to foster job creation, social inclusion, and competitiveness. Policy 1: Strengthen data collection to evaluate the impact of bootcamps Conducting in-depth statistical analysis on prebootcamp and postbootcamp data collected through surveys and focus groups allows inferences to be drawn on the real versus initially envisaged learning and employability impact of bootcamps. In addition, this makes it possible to measure their impact compared to other types of active labor market interventions in the field of IT education (for example, short-term certificate programs, personalized online programs, and so on). If policy makers invest significantly more in catalyzing and/or directly attaining a deeper understanding of skills gaps, they could reduce the burden of market research on bootcamp providers. In turn, data allows providers to adapt and customize their curricula to the most in-demand skills (for example, programming languages, soft skills, and so on). Finally, publicly available data on wages and occupations before and after coding bootcamps can further promote their utility by enabling prospective applicants and providers alike to make informed choices. Policy 2: Link bootcamps to formal education Partnerships between formal education institutions and bootcamp providers pave the way for complementary training on a larger scale. Results from a randomized controlled trial in Colombia show that bootcamp graduates that attended university, in particular, have the best employment outcomes. To this end, policy makers may consider incentivizing formal educational institutions to adapt their curricula and teaching methods to better respond to labor market demand, applying applicable lessons learned and methodology from bootcamps. The current pace of technological changes dictates that the practical application of technologies for problem solving and development of analytical and creative thinking should be encouraged as early as in elementary school. Traditional and theory-based approaches to teaching languages and science, technology, engineering, art and mathematics (STEAM) are causing acute problems, and need to be more practically connected with modern day software and hardware technologies. Furthermore, encouraging elementary school students to use technology can widen the pipeline of potential university students in engineering and mathematics. Entrepreneurship, creativity, and innovation should be equally inspired from an early age, arousing future workers’ motivation to employ technological tools for tackling local issues in their society and environment. A good example of how the love for technology could be nurtured organically from secondary to high school is outlined in the case of Medellin, Colombia (see Box 1). 5 BOX 1: Ruta N’s Horizontes Strategy (Estrategia Horizontes) in Medellín, Colombia The objective of the Horizontes Strategy is to strengthen the vocation for engineering, applied sciences, and art in the youth of Medellin. Its main activity lies in guiding adolescents towards professions that are considered essential to the development of a knowledge society in the city’s and broader digital economy. This initiative emerged in response to a growing digital skills gap in Colombia: by 2019, the country will face a deficit of 90,000 ICT professionals. The initiative is run by Ruta N Corporation, a public joint venture between the Mayor’s Office of Medellin, UNE Telco (UNE), and public utility service company Empresas Públicas de Medellin (EPM), in close coordination of the Ministry of Education of Medellin, and in partnership with the Children’s University EAFIT, Parque Explora, interactive learning center, and Pygmalion, a technology education project at the Software Technology Park of Medellin. The strategy targets grade 6-11 students (aged 11-17) who attend public schools. Horizontes presents an extracurricular activity, which teaches STEAM (science, technology, engineering, art and mathematics) subjects without grading and in a fun, friendly, and unconventional way. In addition, Horizontes seeks to increase the educational and employment aspirations of young people by fostering teamwork, perseverance, critical thinking, and problem solving skills, so that they can be equally inspired in their personal lives. Innobótica, Ingeniería N, and Interchange are the projects that form the Horizontes Strategy. They aim to create, week by week, a favorable environment in which young people are able to develop their technical and social skills to meet the challenges of the 21st century. Innobótica does so by promoting robotics, home automation, and computational thinking; Ingeniería N promotes critical thinking and local problem solving utilizing engineering tools; and Interchange fosters experimentation and networking with leading science researchers, with the emphasis on nanotechnology. Since its foundation in 2014, the program has involved 106 schools, directly generated 106 jobs, and, most importantly, benefited 3,300 young people. Dozens of social innovation projects have been generated by beneficiaries with the help of their teachers. Fifty international researchers have joined the initiative. Source: Ruta N Corporation. Policy 3: Increase access to bootcamps Policy makers can also consider providing funding to increase access to bootcamps for low-income and other vulnerable groups. Given the cost structure of entry-level coding bootcamps in particular, such groups are often under-represented in the intake because of their inability to afford the course. Applicable policy interventions include subsidies towards the tuition fees of low-income applicants as well as other financial support for bootcamps to more effectively reach, retain, and support such population segments (see Box 2). 6 BOX 2: Public support for coding bootcamps In October 2015, the U.S. Department of Education announced the pilot Educational Quality through Innovation Partnerships (EQUIP) program, which essentially qualifies bootcamp students for state-guaranteed loans. Accredited universities can now partner with bootcamps to deliver innovative training programs. Improved access to student funding is expected to fuel further expansion of bootcamps, while universities stand to benefit from innovative programs that are teaching highly relevant skills for the workplace. Tech Hire, launched in March 2015 by the White House, is another high-level initiative that aims to train more middle-class workers rapidly for jobs in technology. Tech Hire partners with bootcamp providers such as Dev Bootcamp and Hack Reactor to provide discounted training slots for underprivileged communities, while developing a wide network of employers open to nontraditional hiring (for example, students without formal degrees). In Kenya, where student loans have only recently been introduced for technical training but have a capped upper limit that falls below bootcamp fees, the has experimented with low-interest loans tied to employment outcomes as well as an instalment structure to make the training course more affordable to lower-income students. However, this requires increased risk and shifts in their business model and new fundraising efforts targeting public and private funds. In Lebanon, SE Factory raises funds to reduce bootcamp fees, and World Tech Makers in Medellín has been supported by the World Bank and Ruta N to offer high subsidies for trainees from low-income segments, which has enabled a significant increase in participation from income strata 1 and 2 of the population. Sources: U.S. Department of Education. 2015; The White House. Once the market opportunity has been demonstrated, and sufficient interest and capacity for the expansion of coding bootcamps has been generated, they can be sustained by other ecosystem players. Public sector interventions are typically important to attract private sector involvement and to create public goods, while the private sector steps in to escalate the number of products and services generated and customers reached. It is through competition in the private sector that the most innovative bootcamp models may emerge, with service offerings catering to the majority of needs at an acceptable cost. For example, the World Bank sponsored a bootcamp for low-income youth in Medellín that has catalyzed the emergence of several new coding bootcamps by private IT education providers. As soon as the private sector sensed an untapped potential for intervention and calculated that the projected benefits outweighed potential costs, they eagerly stepped in, alleviating the pressure on the donor and public sector. 7 Policy 4: Leverage private-public partnerships and networks The partnership between the city of Medellín, Ruta N Corporation, and World Tech Makers illustrates the importance of increased collaboration to conduct rigorous demand analysis and identification of key skills gap. With a view to supporting Medellín to become a regional knowledge and innovation leader, Ruta N is mandated to attract and facilitate the entry of companies in the fields of science, technology, and innovation through its Landing Program. This program aims to attract and integrate companies in the city’s innovation ecosystem through advisory services on taxation and legal matters, as well as access to talent. In 2015, to identify the most in-demand IT skills in the local context, Ruta N carried out a labor market diagnostic with Adecco, one of the largest staffing companies in the world. The assessment revealed that Medellín lacked 2,212 ICT professionals, mostly technicians who could work as junior developers in the software industry. As a result, Ruta N launched a competitive intelligence process that determined that finishing schools3 and coding bootcamps would be the channels through which the city could attain the necessary mass and quality of IT-skilled professionals in a short period of time. The diagnostic results prompted the city to conduct an initial pilot in February 2016 with 23 students. In May 2016, with support from the World Bank, it was expanded to a total of 120 students who eventually participated in a randomized control trial to assess the program’s impact on local youth employment. Networks are also critical to boost relevance, access, and employment outcomes of bootcamps. A cost- effective way to foster a network among bootcamp participants, their trainers, mentors, and academic or private sector partners and clients, is to set up a mailing list and a social media group (for example, Facebook, LinkedIn) for alumni, enabling program participants to exchange relevant educational and job-related updates. Local government can support the creation and growth of such networks through occasional events among bootcamps and industry representatives. The companies that are particularly engaged in this process and other ICT initiatives could be encouraged to form a network through which the public sector could occasionally share the database of coding bootcamp graduates or graduates of other public sector financed training programs. Another policy action would be to set up an advisory group comprised of ecosystem representatives that could independently monitor and advise the management of the coding bootcamp initiative, as well as contribute strategic inputs to other relevant public-sector ICT initiatives and realization of government policies and programs. 8 STEP BY STEP BOOTCAMP DESIGN NEEDS ASSESSMENT Organization of a successful coding bootcamp requires pragmatism for the right goal-setting, endurance for a significant amount of preparatory work, agility during implementation, and a forward- looking vision on how to sustain and scale positive bootcamp results. The rationale for public intervention to support coding bootcamps needs to be based on tackling market failures paired with issue-driven, realistic, and measurable goals for the establishment of a coding bootcamp. From the public-sector point of view, it is important to confirm the existence of market failures and confirm whether there is a need for the use of public resources to resolve said failures. A public-sector intervention should be initiated if market forces cannot address the unfavorable situation on their own. The objective for public intervention should have an analytical foundation that employs objective and diverse data as evidence that (i) a digital skills supply and demand mismatch exists, and (ii) that fixing the digital skills shortages may result in positive outcomes, for example, through improved employment, increased labor productivity, even economic growth and investment attraction. The acquisition of the required data to inform the rationale and development of a bootcamp is challenging. Some pertinent data and evidence can be usually gleaned from the published information and reports of specialized labor agencies, ICT industry associations, specialized academic, policy, and other institutions. Developing a bootcamp also requires quantitative research to determine the exact nature of the digital skills demand and supply. By employing surveys and interviews with the local technical ecosystem stakeholders – such as ICT companies, industry associations, civil society groups, employment centers, and academia – an organization that wishes to design a bootcamp can familiarize deepen its understanding of the issue at stake, and also establish, or, in some cases, re-enforce the relationships with key ecosystem stakeholders. It should be emphasized that the majority of coding bootcamps achieve their goals only if the trained beneficiaries are hired; therefore, it is important to confirm that there are companies willing to absorb this labor, and whether those companies are domestic or international. The examination of the rationale for intervention may last from as little as one month to several months, depending on the data availability, data holding agencies’ willingness to cooperate, and the maturity of the local tech innovation ecosystem. Although there is no “gold standard,” it should be noted that a well-substantiated rationale, which is based on quantifiable data rather than anecdotal evidence, is a critical first step before devising a coding bootcamp implementation strategy. Once the rationale has been established, policy makers can more actively engage key stakeholders and boost the enabling environment for a bootcamp to enter the market. They should examine targets for the bootcamp, and assess organizational, human, and financial needs for successful implementation. Different implementation models can be envisaged, including outsourcing to a specific training provider or supporting an already existing coding bootcamp provider through cost sharing or in-kind support (for example, access to facilities, its staff, equipment, connectivity, and so on). 9 Even if a bootcamp provider does not initially have a clear idea of an implementation model, it is important that it assigns competent staff to drive the initiative forward. Staff have to be able to advocate for a coding bootcamp as part of the wider strategy of their organizations, have hands-on experience in project management (preferably, in similar training projects), and be good at building and maintaining mutually beneficial relationships with internal as well as external stakeholders. The exact division of responsibilities among staff is to be determined based on the mission, mandate, organization’s corporate (internal) culture, and requirements of the project. It should be noted that implementation of a coding bootcamp requires a high degree of multitasking and teamwork, compared to mere bootcamp supervision or partner support. NEEDS ASSESSMENT CHECKLIST ⃝⃝ Conduct desk and field research to gain an understanding of the digital skills supply and demand mismatch. ⃝⃝ Develop analysis and rationale for intervention based on gathered data. The coding bootcamp should be based on a real need, not a perceived one. ⃝⃝ Formulate an implementation strategy that accounts for available organizational, human, and financial resources. ⃝⃝ Put together a high-performing implementation team which reflects appropriate mix of technical, project management, and stakeholder engagement skills. 10 SELECTION OF TARGET BENEFICIARY GROUP AND OUTREACH STRATEGY Thanks to background research, its mission and mandate, a bootcamp provider should have a clear idea on the target beneficiary group. In some cases, the rationale may suggest reaching out to groups with different defining characteristics (for example, sociodemographic, educational, and so on) to test how these may influence the learning and employability outcomes of one group compared to others. This is usually important for fundraising and project scaleup, and it is not uncommon to find coding bootcamp providers subsidizing a certain number of training spots for low-income students or women. Most often, coding bootcamps target young people with at least some command of the English language. Some bootcamps, especially those reporting very high employability outcomes (for example, over 90 percent), usually require prior IT knowledge. Regardless of the target beneficiary group, it is a good practice to try to select an equal sample of different profiles of beneficiaries who are affected by the digital skills mismatch: for example, students, unemployed graduates, unemployed with no university education, women, minorities, and so on. Such a distribution will prove to be particularly useful at the evaluation stage. The application process is similar across bootcamps, relying on short online applications, interviews, and, in some cases, on basic exams or character-revealing exercises. Typically, interested applicants are invited to fill out a short online questionnaire providing basic personal information on education, interests, employment, and so on, and outlining their interest and motivation for the training. Then, eligible applicants have to take basic exams, tests, or creative challenges. In some instances, applicants may be required to undertake additional coursework prior to taking the basic exam; this is to ensure that they are on par with other candidates in terms of their prior coding knowledge once the bootcamp starts. Finally, those who succeed are invited for an interview: either face-to-face or via a telephone or online interview. Interestingly, bootcamp providers have been quite active proponents of diversity in the tech industry (see Box 3). BOX 3: Typical successful applicant profile across U.S. and Canadian coding bootcamps Industry monitor, Course Report (2016), analyzed graduates of U.S. and Canadian coding bootcamps in 2016. It reported the following insights with regards to the application process. The mean age of bootcampers was 30, with 56.7 percent being male and 43.3 percent female; 60.8 percent had a bachelor degree, 15.6 percent above, and 23.6 percent below bachelor level; 40.4 percent had no prior programming experience, 47.3 percent had some in their free time, 10.9 percent had some at work, and fewer than 1.5 percent had a programming background on a full-time basis in their work. A reason for attending a bootcamp for over 89 percent of applicants was the prospect of getting a programming job. The most cited reason (4.3 percent) was starting a company. Before enrolling in a bootcamp, over a quarter of applicants earned less than US$25,000, that is, were low-income (compared to 8.7 percent who earned more than US$75,000, the highest reported salary level). Source: Course Report 2016a. 11 To reach out to the identified target beneficiary group, bootcamp providers usually rely on mass media, digital marketing (websites and social media), information sessions at schools, universities, conferences or networking events, and on word-of-mouth. They place information and paid advertisements on the Internet several weeks in advance of the screening process. Partners from the ecosystem and outside of it (for example, community leaders) are asked to join forces and cross-post a call for applications. Engaging partners for outreach is particularly important if a bootcamp provider is new to the IT education scene. External communication is key not only to the application process but to the entire training cycle. Raising awareness of a coding bootcamp, its specific programming, and promoting success stories of its beneficiaries is indispensable for a positive reputation of a bootcamp provider, its external relationship-building and fundraising efforts. Frequent and consistent communications can help counter common stereotypes, and demonstrate the value of such programs, even if they are not part of the formal education system. BENEFICIARY SELECTION CHECKLIST ⃝⃝ Determine target beneficiary group based on background research on the issues at stake, the organization’s mission and mandate. ⃝⃝ Consider selecting an equal sample of different beneficiaries’ profiles to enable a more credible postbootcamp evaluation of learning/employability outcomes. ⃝⃝ Advertise bootcamp and application through multiple channels, including mass media, social media, information sessions, and word-of-mouth. ⃝⃝ Secure support of local ecosystem and external partners for cross-outreach. ⃝⃝ Invest in external communications throughout the entire project cycle – inform, convince, and engage. 12 SELECTION OF OPTIMAL MODEL AND PROVIDER In their desk and field research on some 40 bootcamp providers in Africa, Asia, Latin America, the United States, and Europe, the International Telecommunication Union (ITU) conceptualized four broad models dominating the coding bootcamp market (International Telecommunication Union 2016): 1. Ready-to-Work model: This is the traditional approach to coding bootcamps – intensive 12 to 24 weeks full or part-time rapid skills training programs that prepare people to qualify for employment shortly after the training ends. 2. Bootcamp+ model: This is an extended training approach – longer training programs (one to two years) that equip students with a broader range of sustainable income-generation skills in addition to coding competencies (for example entrepreneurship). 3. Mini Bootcamp model: These are very short-term training programs ranging in length from two days to one month. They are typically designed to spark interest in learning the basics of programming, to recruit or identify talent, for professionals to update their skills, and for outreach and community building. 4. Early Education model: These are efforts to trigger interest in programming at an early age. This model includes workshops, hackathons, and online platforms as well as more encompassing efforts such as schools integrating coding skills into their curriculum. Although not focused on employability in the short term, the early education model is an important trend to monitor. Although this toolkit is dedicated primarily to the Ready-to-Work model, which is becoming known as a “standard” coding bootcamp, the other three models also provide useful lessons, particularly in the context of devising medium- and long-term strategies for IT education. Their outcomes differ because of the differences in the coding bootcamp programming and duration of the skills attainment process. Some bootcamp providers are starting to experiment with more than one model of training to test which one yields higher learning and employability outcomes. Some providers go as far as to offer training models on-site (in physical locations) at the same time as online (virtual bootcamps). A training model stems from the ultimate goal of the bootcamp provider (for example, knowledge/ skills attainment versus employability enhancement) and its available organizational resources. If the ultimate goal is to accelerate job creation in the ICT sector, then the first two models (Ready-to-Work and Bootcamp+) are the most appropriate. Similarly, if the coding bootcamp intends to generate new knowledge and interest in ICT, Mini Bootcamp and Early Education models would be key. Which model to choose should be also driven by a coding bootcamp’s business model4. Ready-to- Work, Bootcamp+, and Mini Bootcamp models are more amenable to commercialization, and hence more appropriate for profit-maximizing entities, than, say, an Early Education Model, which is oriented at preschool and primary school children and their parents. A bootcamp provider running an intensive Ready-to-Work program needs to have the most substantial financial capacity, as it has to perform the highest amount of preparatory work in the shortest timeframe. In a similar vein, an organization interested in an Early Education model should budget appropriate resources for long-term advocacy and multistakeholder consensus-building required by amendments to educational curricula. A qualified bootcamp provider needs to demonstrate capability in design and implementation of this or a similar type of training, as evidenced by the entity’s relevant experience, and qualifications of its professional staff. It needs to have a ready-to-deploy training curriculum, dedicated staff and local trainers, an outreach and screening methodology, and a strategy for public relations and fundraising, as needed. At the same time, the provider should have knowledge of principles and methods of impact assessment to be able to carry out or support project monitoring and evaluation. 13 The provider has to be deeply embedded in the local tech innovation ecosystem, or at least have sound knowledge of its dynamics, including personal exposure to its key players. This level of exposure determines one’s ability to rally support for the bootcamp among prospective employers of bootcamp graduates. The provider needs to have access to the training facility with acceptable conditions (see Venue for a Bootcamp). Additionally, a provider with a proven and tested online platform to manage trainees and provide e-learning solutions should be at an advantage in the selection process. Such a platform helps the provider more easily identify risks, and monitor and report on the beneficiaries’ and instructors’ performance. It also allows students to continue their coding work once the on-site classes are over for the day. Last but not least, a bootcamp provider needs to have sufficient financial capacity to pay for a number of capital (for example, venue) and operational expenses upfront and be reimbursed on a retroactive basis. It is good practice for the bootcamp provider to allocate a certain percentage of funds for contingency reasons. Unforeseen incidental expenses can result from a variety of circumstances, such as the need to procure lacking furniture or equipment; secure a new venue to accommodate the bootcamp; or establish higher security arrangements (for example, hiring of security personnel, running shuttle services, and so on). MODEL AND PROVIDER SELECTION CHECKLIST ⃝⃝ Pick the most appropriate of the four most popular models (Ready-to-Work, Bootcamp+, Mini Bootcamp, and Early Education). ⃝⃝ Select bootcamp provider based on: o Track record of designing and implementing similar type of training o Methodology and learning tools (training curriculum, outreach and screening, staff competencies, and so on) o Depth of engagement with local tech innovation ecosystem, as evidenced by connections to key players o Availability and/or access to training facility with acceptable conditions o Use of a proven and tested online platform to manage trainees and provide e-learning solutions o Financial capacity to cover upfront expenses and agree to retroactive reimbursement o Evidence of contingency funding to cover for incidental expenses. VENUE FOR A BOOTCAMP To execute the planned training activities, a bootcamp provider can either use its own premises or rent them. If providers hire computer classrooms from, for example, coworking spaces, Internet cafes, or universities, they need to ensure that other users of the host’s IT equipment do not have access to 14 the files and materials used or produced by bootcamp students. Bootcamp trainees should be able to practice in the same venue before or after classes. Having a backup option is always preferable, should any issues arise, once the training begins. A “rule of a thumb” is to use a centrally located venue, in a safe neighborhood with good access to public transport. The latter provides students with the opportunity to network and attend events or meetings with the larger IT community. Training classrooms should not suffer from external noise, since the learning process of acquiring IT skills requires a high amount of concentration. At a minimum, the venue should have an appropriate classroom furniture (for example, necessary number of desks and chairs, a black/whiteboard), adequate IT equipment (for example, computers with a set of minimum requirements, keyboards, headsets, projectors), instant access to electricity, heating or air-conditioning, high-speed broadband Internet connectivity, and emergency equipment (for example, fire extinguisher, first aid kit). The venue should be wheelchair accessible and have appropriate accommodation for people with other types of disabilities. It is good practice that coding bootcamp instructors participate in the search for a classroom as they can immediately evaluate whether the equipment is adequate. A venue located in a remote area or with high criminality rates may require security on-site or proper transportation arrangements (for example, organization of shuttle services). It may be advisable for a bootcamp provider to establish a direct channel of communication with the neighborhood community (for example, with community leaders, activist groups, neighborhood organizations, and so on) to inform residents of the bootcamp taking place in the vicinity of their homes. Bootcamp trainers and participants should always be aware of the location of the closest health facilities (for example, clinics, hospitals) should any emergency arise. It is highly recommended that the trainers possess basic first aid skills to be able to take appropriate measures in such cases. In addition, acquiring medical insurance cover for the students is highly recommended. VENUE ARRANGEMENT CHECKLIST ⃝⃝ Determine whether training can be conducted in-house or if external venue needs to be rented. ⃝⃝ Reserve facility accordingly and make sure to have a back-up option. ⃝⃝ Select training facility based on: o Proximity to city center, in a safe neighborhood and with good access to public transport o Adherence to a minimum requirements technical checklist with regards to furniture and appliances, basic infrastructure, computer equipment, and Internet o Access to on-site security or shuttle services, where needed, and capacity to liaise with neighborhood community. ⃝⃝ Inform trainers and participants about the location of the nearest health facilities. ⃝⃝ Instruct trainers on first aid basics, and if possible, get health insurance for participants. 15 PARTNERSHIPS WITH LOCAL ECOSYSTEM PLAYERS Regardless of whether a provider is a bootcamp executer, supervisor, or partner, it must establish mutually beneficial partnerships with key players in the local technology innovation ecosystem to achieve its strategic goals. What is a tech innovation ecosystem? Urban technology innovation ecosystems are defined as the collection of stakeholders, assets, and their interactions resulting in technology (in particular ICT)-based innovation and entrepreneurship. Cities are increasingly emerging as the new centers of technological innovation. A shift is under way from technology parks in suburban areas, where universities, research labs, and the private sector are located together, to entrepreneurial activity within cities. Factors such as proximity, density, and variety of people and firms contribute to this phenomenon (Mulas and others 2015). Forging partnerships with large IT corporations or small and medium sized enterprises (SMEs) is essential, especially if the goal of the bootcamp is to help beneficiaries find jobs upon graduation. These IT companies may participate in the bootcamp as trainers or guest speakers, provide mentorship to beneficiaries, or engage in similar ways (for example, engage in student sponsorships, provide a training facility, and so on). After the bootcamp is over, these companies can hire beneficiaries or provide them with internships. In some instances, companies can sponsor students’ fees, the graduation ceremony for beneficiaries, or networking events postgraduation. It is also important to establish fruitful collaboration with academic institutions and civil society (for example, industry associations, IT education providers, NGOs, and foundations). The former can assist with data collection to scope the need for a coding bootcamp, support the process of curriculum development/adaptation, support training delivery, and outreach among students. The latter could become indispensable partners on outreach and event planning. They could also be a significant source of applicants. Finally, it is equally important to garner support from government entities mandated to oversee or support the ICT sector, innovation policy, and active labor market interventions. Their support may come in the form of assigned labor to implement the bootcamp (for example, advisers), allocation of a training venue or equipment, or support with outreach and communications. Where necessary, memoranda of understanding (MoU) should be signed to formalize such partnerships. PARTNERSHIP CHECKLIST ⃝⃝ Forge partnerships with the local tech innovation ecosystem for the bootcamp design and implementation. ⃝⃝ Develop cooperation mechanisms with relevant government entities, academic institutions and civil society for data collection, curriculum development, training delivery, or outreach support. ⃝⃝ When needed, sign an MoU to formalize a partnership. ⃝⃝ Work with ecosystem partners in a communications and engagement campaign. 16 BUSINESS MODELS As types of coding bootcamps have evolved, so have the options for their financing. In the United States and Canada, coding bootcamp providers are typically for-profit companies. Their revenues rely heavily on the ability to secure job placements for graduates. On average, most bootcamps charge a fixed tuition fee in the amount of US$11,451, with a range of US$5,000-20,000, for a 13-week program (Course Report 2016b). High fees are justified by the outstanding employment track record and postbootcamp salaries (Course Report 2016a, Table 10). To attend to those with lack of or poor credit history, U.S. and Canadian bootcamp providers are increasingly turning to alternative lenders. Student loans are offered through these third parties on favorable terms, ensuring that upfront payments are released to bootcamp providers, guaranteeing a constantly positive cash flow. Some coding bootcamps (for example, App Academy) do not charge any tuition fee at all, but rather take a fixed percentage of postbootcamp salary within the first year of employment. Others rely on partner employers for tuition fees: for example, bootcamp participants are exempt from fees in case they are employed by one of the partner firms. In some cases, in-house scholarships are offered to partially offset tuition costs and/or living expenses. In others, bootcamp providers make arrangements with other entities (for example, government) to provide subsidized training for specific vulnerable groups (for example, U.S. veterans, prisoners, low-income populations). Other coding bootcamps provide differentiated services that include both entry-level programming bootcamps and more advanced-level courses. They are also expanding into the provision of job placement for which they charge employers a fee. Through these varied revenue streams, they are able to cross-subsidize the relatively high tuition fees for entry-level courses, for which prospective employers are unlikely to pay, in general. This is particularly relevant for emerging markets where there are insufficient student loan and other student financing mechanisms. Available data for coding bootcamps located outside high-income markets suggest that these programs are implemented by for-profits, nonprofits and social enterprises (see Box 5 for typical sources of funding). Typically, coding bootcamp tuition cost ranges from US$500-2,500 in Africa, and US$1,500-3,000 in Latin America. In Asia, where franchises of U.S. bootcamps are taking root, the cost of training can reach as high as US$10,000 (International Telecommunication Union 2016). In Kenya, Moringa School operates as a for-profit firm that aims to attract high-potential students irrespective of their socioeconomic background. It relies on several revenue sources, namely tuition (US$500-1,250), school training (US$150 per two-week course), and job placement fees. With four locations in Latin America, Laboratoria relies on deep relationships with partner employers as a nonprofit social venture. Students start paying for the 24-month program after graduation and only once they find a job. Hence, Laboratoria’s ultimate goal is to find a personalized match between a student and an employer, embedding postgraduation monitoring of students in their operating model. A public-private partnership (PPP) model can be used to establish a bootcamp, especially in places with a high concentration of the poor and a vibrant local private sector, for example, the bootcamp organized in Medellin, Colombia, under the Decoding Bootcamps initiative. The World Bank, Ruta N Corporation5 and the Municipality of Medellin contributed funding and in-kind resources for the bootcamp targeting low-income youth, provided by World Tech Makers (WTM). WTM is a for-profit that charges tuition ranging from US$2,000 in Colombia to $3,500 in Brazil. The provider also offers online training programs for between US$14.99 and US$499.99 per month. Following this initiative, several independent private sector coding bootcamp providers entered Medellin to start their own courses. This model seems to be a particularly good fit for the burgeoning IT innovation hubs outside high- income economies. 17 BOX 4: Typical sources of funding for coding bootcamps The ITU report on bootcamps has discerned the following typical sources of funding. Typically, more than one revenue stream is being used to sustain bootcamp operations. 1. Student tuition: Fees students pay to enroll in the program. The amount varies depending on the organization and type of training. 2. Student contribution to the organization after they are employed: A percentage of a student’s monthly salary is paid to the bootcamp provider for a certain period of time while they are employed. This is both a revenue stream and a funding strategy for many bootcamp providers to ease the burden of advance tuition payment. 3. Employer hiring fees: Fees charged to companies when they hire one of the bootcamp graduates. 4. Startup accelerators: Bootcamp provider helps students create a startup and keeps a percentage of the startup holdings. 5. In-house recruiting: Provider has or creates a recruitment or web development agency to find prospective projects on which graduates may work. 6. Licensing curriculum: Bootcamp provider designs a high-quality curriculum and then licenses it for a fee. 7. Donations [cash or in-kind]: This is a particularly important funding source for nonprofit organizations offering coding bootcamp training. The type of donation and the activities that donations cover vary from organization to organization. In some instances, in-kind donations are in the form of a physical space, Internet access, training space, or payment of utilities. Cash donations are often used as seed investment to start the training program, expand the training to more people or provide additional types of training, or to sponsor student enrollment fees. Source: Adapted from International Telecommunication Union 2016. 18 Organizations with strong links to the local tech innovation ecosystem can experiment with revolving funds and student sponsorship to cover operational expenses. The former are set up to allow bootcamp graduates to contribute, once they have been hired by a company, to cover the expenses of new students. They could be particularly well suited to the environments with developed financial systems. Student sponsorship relies on attracting private ICT companies to sponsor talented young people to study in a bootcamp and then hire them as full-time employees. Places with severe shortages in digital skills and high supply of IT jobs could especially benefit from such sponsorship. Student sponsorship can cover not only tuition fees but also participants’ access to basic self-study resources (for example, home computers, broadband internet). Basic subsistence expenses, such as transportation and meals, can also be covered if costs are prohibitive. Access to training resources is indispensable to positive learning outcomes. Hence bootcamp providers need to ensure that the lack of access to a personal computer/laptop and/or a working broadband internet connection does not impede one’s learning progress. The sponsorship scheme can either rely on grants/stipends (nonreimbursable) or microloans (reimbursable but on favorable terms). Where coding bootcamps are largely or fully subsidized, one cannot stress enough the importance of requiring participants to make at least a symbolic payment. Since this type of training has a high attrition rate, a minimal payment helps to ensure beneficiaries’ commitment and warrant against no- reason dropouts. BUSINESS MODEL CHECKLIST ⃝⃝ Determine if providing a subsidy would align with and/or enable bootcamp’s strategic goals. ⃝⃝ Evaluate best financing option for a bootcamp provider, depending on business model, mission, mandate, and bootcamp goals. Specific schemes to consider include: o A PPP financing model could be applicable in areas with a high concentration of the poor and a vibrant local private sector o Revolving funds in environments with developed financial systems o Student sponsorship that can cover tuition, access to training resources, and/or basic subsistence expenses in areas with high shortages in digital skills and high supply of IT jobs. ⃝⃝ Where coding bootcamps are largely or fully subsidized, charge a small, symbolic fee to warrant against no-reason dropouts 19 IMPLEMENTATION LEARNING PROCESS Onboarding is equally important for students admitted to the bootcamp and staff (instructors). Students need to understand the requirements of course attendance and performance in advance. Trainers should be instructed on the course syllabus, teaching requirements, as well as methods to properly attend to the group and individual needs, marrying pedagogical competencies with technical expertise. The course syllabus and bootcamp program conditions are usually explained to students on the first day of training, which is sometimes turned into a launch event with external stakeholder presence (for example, partners, donors, and media). Aiming to create a level playing field between students with no prior coding knowledge and their more advanced peers, some coding bootcamps institute compulsory prebootcamp preparation. For example, U.S.-based Hack Reactor requires 50 to 70 hours of such preparation, while Kenyan Moringa School has developed a one-month preparatory onsite course. If the coding bootcamp does not cover the absolute basics, instructing students to master certain subjects on their own, before the training kicks off, is reasonable and pragmatic. This step may filter out those students who would later find themselves less motivated or capable of continuing with the training. Another aspect of the onboarding process is to secure the commitment of students to dedicate fully to the training, especially if a bootcamp is partially or fully subsidized. It has been noted that the attrition rate for partially or fully subsidized bootcamps is usually higher, since the cost of dropping out is lower. To minimize the risks of dropouts, some bootcamp providers ask all of their students to sign a commitment letter, outlining students’ rights and responsibilities. While such a letter may not have any binding legal power, it serves as a proof of an arrangement made, which should be respected by both parties. Onboarding for instructors, especially those who have little teaching experience, should involve coaching on the most successful pedagogical methods used in rapid tech skills training. Trainers with little relevant pedagogical experience should take relevant courses, participate in workshops, read specialized literature, and so on. Certain gamification aspects – from point scoring to encouraging competition among students – are incorporated into the learning process in order to drive engagement and increase the spirit of competition among the cohort. It is equally important to involve instructors in the design or review of the bootcamp curriculum and teaching resources so that they master the teaching content and resources. Overall, it is good practice to organize a course syllabus according to the modules that address simpler topics at the beginning and gradually progress toward more complex ones. Such coursework planning will guide the process that both the instructors and the students will develop, in addition to enabling assessment processes to be carried out according to previously defined goals. 20 LEARNING PROCESS CHECKLIST ⃝⃝ Onboard the students with a clear explanation of the requirements of course attendance and individual performance. Make students sign a commitment letter to prevent no- reason dropouts. ⃝⃝ If the coding bootcamp does not cover the absolute basics, instruct students to master certain subjects on their own before the training starts. ⃝⃝ Involve instructors in curriculum design and/or review to ensure they are fluent in the course content and make good use of teaching resources. ⃝⃝ Provide resources to master their pedagogical competencies. ⃝⃝ Organize a course syllabus according to modules that address simpler topics at the beginning and gradually progress towards more complex ones. CURRICULUM DESIGN The curriculum of coding bootcamps differs based on the goals, target demographics, and the nature of the current market demand for digital skills. Usually the bootcamp curriculum is subject to reviews and modifications based on industry inputs. Bootcamp providers survey local ecosystem stakeholders – from SMEs to multinational companies to industry associations – to understand their existing and projected demand for certain types of skills and programming languages. At the same time, bootcamp providers make their assumptions of which IT subjects or programming languages are in demand based on publicly available online sources (for example, real-time job matching websites) and private market intelligence data. The technical content of coding bootcamps is usually based on international best practice coursework, for example, internationally acclaimed proprietary and free online courses, video tutorials, massive open online courses (MOOCs), and so on), but with customization based on local IT industry needs and peculiarities of local culture. Tech skills curricula have become more diverse over the past few years, as course offerings have expanded from basic web and mobile development to more complex IT subjects. Still, full-stack web development is the most widespread program offered, teaching skills related to server, network and hosting environment, relational and nonrelational databases, application programming interfaces (APIs), user experience, and project management. A standard coding bootcamp curriculum covers programming fundamentals, such as working with database modelling and object relational mapping (ORM), understanding model-view-controller (MVC) frameworks, and executing application deployment. The World Bank’s case studies from Colombia, Kenya, and Lebanon demonstrate that Javascript was the most common programming language in 2016, and the majority of the technical content is dedicated to front-end web development, that is, CSS and HTML. Industry monitor Course Report, attests that in the U.S. and Canada, Javascript has recently overtaken Ruby on Rails (object-oriented general-purpose programming language) as the most popular language; however the number of graduates proficient in Ruby still outnumbers those in Javascript (Course Report 2016b). 21 Some coding bootcamps focus exclusively on teaching specific technical skills, whereas many also have coursework covering socioemotional skills, such as business communication, problem solving, or portfolio management. This seems to be particularly relevant in transitioning economies, where the formal education system faces a “disconnect” from the labor market, and the business culture is only just taking root. Socioemotional skills seem to be most extensively covered by those bootcamps that cater to marginalized populations, such as women, disenfranchised youth, and the poor. BOX 5: Links between the skills demand and the bootcamp curriculum Strategies to assess the current and expected skills needs of the industry, and thus develop the curriculum, vary from bootcamp to bootcamp. In the cases of World Tech Makers (WTM) in Medellín and SE Factory Beirut, which were part of the Decoding Bootcamps activity, two completely different methods were used. Medellín: For the past decade, and in line with national policies, the city of Medellín has developed its own programs to strengthen the IT industry. In particular, Ruta N Corporation, the innovation arm of the municipal government, has the mandate to attract companies in the fields of science, technology, and innovation. The combination of local and national policies and Ruta N’s work resulted in the growth of the software industry in the city, which also led to a growing demand for IT professionals. To assess the skills demand in the IT sector, Ruta N and Adecco, one of the largest staffing companies in the world, carried out a labor market diagnostic in 2015. This study identified the most in-demand skills and highlighted those that were scarce among the local labor force. The overarching conclusion of this assessment was that Medellín had a gap of 2,212 professionals in the IT field, among them, many junior developers. This analysis prompted Ruta N to design appropriate strategies to address this gap. One of these strategies was to attract coding bootcamps to the city to rapidly boost the pipeline of junior developers with the specific skills that the industry required. WTM, a bootcamp that was already operational elsewhere in the country, demonstrated a methodology that would prepare its graduates to enter a formal working environment at an IT company.Together with municipal government of Medellín, the World Bank offered WTM financial incentives in the form of tuition subsidies to expand to the city and take part in the research pilot under the Decoding Bootcamps initiative, targeting low-income youth. Through the needs assessment conducted by the city and following its first pilot in Medellín, it tailored its curriculum to reflect programming language requirements by local industry. Seventy-three percent of the bootcamp students were employed six months after graduation. Strategies to assess the current and expected skills needs of the industry, and thus develop the curriculum, vary from bootcamp to bootcamp. In the cases of World Tech Makers (WTM) in Medellín and SE Factory Beirut, which were part of the Decoding Bootcamps activity, two completely different methods were used. 22 Beirut: In Beirut, SE Factory, the World Bank’s bootcamp partner for the Decoding Bootcamps initiative, conducted its own analysis of the skills demand and, based on this, designed its curriculum. One of the founders of SE Factory also runs a startup accelerator and an investment fund, therefore was also able to leverage his own industry knowledge and network of companies for the needs assessment. SE Factory concluded that most of the startups within their network were looking to hire full-stack web developers who were well-versed in open source languages. In parallel, they examined technology skills development offered by local universities. There are numerous computer science graduates in Lebanon each year, but many of them struggle to find employment in the field. This affects those from lower socioeconomic backgrounds even more acutely, as employers are more reluctant to hire them given the perceived lack of both soft and technical skills. Understanding the skills demand from the industry and the skills gaps among local university graduates, in particular of those from lower socioeconomic backgrounds, had enabled SE Factory to develop a curriculum based on international best practice and adapted to local market needs, so as to support graduates from second- tier academic institutions and lower socioeconomic background. Thereby it supported such students to compete more effectively with their peers from more elite institutions and higher income background. In the Decoding Bootcamps study, 82.6 percent of the graduates were employed six months after the finalization of the bootcamp (Mulas et al 2017). In the above two cases, bootcamps focused on local and or regional demand. However, there are emerging rapid tech skills providers and talent placement organizations, such as Andela, that offer bootcamps focused on high-income markets, and link their curriculum to demand from companies in the United States or the European Union. Bootcamps are also expanding with the introduction of internships and job placement services, for example, the Moringa School or Andela, for which they can charge a fee and introduce an additional revenue stream. This service also strengthens their ability to more effectively capture the demand and build relationships with private sector employers. Bootcamps also organize so-called Demo Days, whereby graduates pitch and demonstrate a project they have worked on during training at the completion of the bootcamp to private sector companies who provide feedback, which impacts hiring and informs methodology and curriculum going forward. Bootcamps also regularly update their curriculum based on industry needs and adjust their methodology to balance quality with scalability to meet increasing demand. 23 The socioemotional skills curriculum structure is similar across bootcamps, although the emphasis and resources placed on career development varies. Students are also assisted with the design of their résumés, cover letter writing, personal professional brand building and management on social media, and assembling of their IT portfolios for prospective employers. Those bootcamps that also have an entrepreneurship focus teach their students how to pitch their ideas publicly to raise funds. The end goal of the socioemotional and job readiness skills curriculum is to build participant confidence that they will become successful in coding after as little as 3-6 months of training. In addition, given the evolution of technologies or coding languages in the market, socioemotional skills can teach students a set of learning methods enabling them to adapt themselves to new languages without having to undertake another bootcamp. In-class, tech and socioemotional skills sessions are usually combined with inspirational talks and workshops from industry specialists. Some bootcamps also offer mentorship opportunities with industry specialists. They aim to help students determine their career path in ICT, increase their personal networks, and ultimately, find jobs. CURRICULUM DESIGN CHECKLIST ⃝⃝ Prepare a market-driven curriculum by collecting inputs from local ecosystem players combined with market data. ⃝⃝ Liaise with IT companies to determine a programming language to be taught during the bootcamp and adapt the language accordingly, as market needs evolve. ⃝⃝ Make socioemotional skills coursework part of the curriculum. It is of particular relevance to some demographic and socioeconomic groups. ⃝⃝ Combine in-class tech and socioemotional skills sessions with talks and workshops from industry specialists, networking sessions, and mentorship. COMMON CHALLENGES The challenges that arise at the training implementation stage usually relate to infrastructure, skills, and internal communication. Even if an appropriate venue has been selected for the bootcamp, basic infrastructural issues may arise, which could impede the learning process. Power cuts and broadband internet connectivity problems are common in rapidly urbanizing cities. A back-up electricity generator and access to more than one broadband network can alleviate these infrastructural pressures. Since bootcamps are homework heavy, it is recommended that minimum technical requirements for personal computers be specified in advance of the training to ensure their compatibility with the software used by the training provider. The applicant screening, no matter how comprehensive, inevitably filters in participants with a varying degree of coding knowledge. Across bootcamps, a common concern is how to make those at the higher end of coding knowledge and much less experienced peers work effectively in tandem on 24 project-based assignments. Individual attention to students – equally to those who are advancing and those who are lagging behind – is the right approach to create a level playing field among bootcamp participants. Having an assistant instructor in everyday classes is also helpful to support students who are lagging behind in certain exercises or to substitute the primary instructor in case of their absence. In addition, the curriculum should cover introductory topics (especially on simple logic and programming logic) during the first weeks of the training to increase students’ level of proficiency with basic concepts and confidence in their ability to succeed. In a similar vein, a rapidly executed screening process can hardly evaluate one’s English language command. Having basic proficiency in this language is essential to programmers, because the codes, links, and exercises are all in English. One of the possible solutions is to offer remedial coursework in parallel to the bootcamp, and to lead all training – even on socioemotional skills – in English to enhance participants’ conversational and technical language skills. If the bootcamp curriculum requires access to the self-paced e-learning platform or solution, the learning expectations should be clearly spelt out at the beginning of the training and appropriate onboarding for the use of the platform/solution should take place. E-learning is still at its early stage; to some it presents a challenge on its own. If participants are not comfortable using innovative online education they will be more likely to drop out. Last, any inconsistencies or distortions in internal communications, flowing from the bootcamp provider to participants and vice versa, negatively impact the learning process. To some participants, intensive bootcamp training causes a great deal of stress and anxiety. Add disruptions in communications and a student may consider dropping out.A bootcamp provider should encourage bottom-up communication and establish feedback loops on the learning process and how it should be improved. At the same time, it should communicate clearly, in plain language, and in a consistent manner about the training phases, what they offer, and how to succeed using available resources. Internal communications should inform, engage, motivate, and inspire, assuring participants that their end goals are within reach day by day. CHALLENGE CHECKLIST ⃝⃝ If a bootcamp venue is likely to face basic infrastructure issues, be ready to secure back-up equipment or scope out other alternative solutions. ⃝⃝ If there are any minimum technical requirements for personal computers to be used for homework, inform participants in advance of the training. ⃝⃝ Pay close individual attention to students and conduct introductory classes on coding basics. An assistant instructor can be helpful to help students that lag behind in certain exercises. ⃝⃝ In the event of a cohort with insufficient English language skills, offer a remedial English training. ⃝⃝ Ensure the cohort is comfortable with the self-paced e-learning platform or solution, if applicable. ⃝⃝ Encourage bottom-up communication and establish appropriate feedback loops. 25 POSTBOOTCAMP ACTIVITIES GRADUATION CEREMONY Completion of a bootcamp presents an opportunity to acknowledge each participant’s willingness to learn coding and increase wider awareness of the training. The most common way to do this is through a graduation ceremony during which participation certificates are distributed in a festive atmosphere, and where project stakeholders are all convened to salute the participants and their trainers. Media advisories and press releases should be prepared at least a week in advance. Some bootcamps combine graduation ceremonies with a Demo Day, which can be a good opportunity for students to pitch publicly the individual projects pursued over the course of the curriculum to prospective employers from the local ecosystem. It is good practice to circulate information about the certificates at the beginning of the training or even at the applications stage. For many participants and local employers, these certificates serve as the only testament of the training undertaken, and hence are treated on par with formal educational diplomas. Clear and consistent internal communication should contain information about a set of minimum requirements to obtain a certificate. Training attendance and learning performance are usually the two factors influencing whether one is entitled or not to a certificate. To monitor attendance, bootcamp providers rely on attendance sheets or automatic logs from their online platforms. Since most of the learning process is project-oriented, completion of projects in a form of class work or home-based assignments is directly correlated with the level of learning performance. GRADUATION CHECKLIST ⃝⃝ Set minimum criteria for issuing participation certificates. Communicate clearly and consistently about the certificates’ issuance process. ⃝⃝ Plan a graduation ceremony that convenes project stakeholders and the media. ⃝⃝ Consider hosting a Demo Day competition combined with a graduation ceremony. PLACEMENT OF GRADUATES AND OTHER TYPES OF POSTBOOTCAMP SUPPORT Unlike other training programs, coding bootcamps typically provide job placement services for their graduates through an effective two-way channel of engagement established with local ecosystem partners, especially private companies. Coding bootcamps pitch their graduates to companies, and companies turn to bootcamps with job announcements. In a way, bootcamp providers act as IT recruiters, offering companies tailored assistance with regards to the candidate search, while tapping into their in-house pool of job seekers. 26 Some bootcamp providers place talent locally and globally (as individual freelancers via online work platforms or directly with foreign companies). For example, Kenyan Moringa School has built strategic partnerships with over 40 hiring partners. In addition, the School closely cooperates with Moringa DevShop, which is one of the companies under the Moringa umbrella (but separate from the school) that can hire graduates to work directly on outsourcing projects for U.S.- and European-based clients. Job matching or placement of this sort may require fees separate from the tuition fee for the coursework. Postbootcamp employment rates reported by training providers are typically moderate-to-high (75 percent+), increasing over time. However, these do not distinguish between the types of employment (full-time position versus nonpaid internship), nor work area (programming versus nonprogramming jobs). On average, 73 percent of graduates of U.S. and Canadian coding bootcamps in 2016 were employed in full-time jobs that required skills learned at a bootcamp. Among those, most were salaried full-time positions and only few were part-time or self-employed. These graduates’ average salary increased by 64 percent, and they experienced an average salary lift of US$26,000 (Course Report 2016a, Table 10). The average payback period on the bootcamp education is about three months, if a graduate is employed6. Despite low availability of comparison data on the bootcamps’ performance outside North America, the ITU established that coding bootcamps falling under the category of a Ready-to-Work model report equally high job placement of their graduates in the range from 60 to 100 percent, depending on the organization (International Telecommunication Union 2016). Based on interviews by the World Bank with bootcamp providers, postbootcamp employment rates range from 70 percent to 100 percent. Employment outcomes vary depending on how selective the bootcamp is in its admissions procedure, the amount and type of resources devoted to career development throughout the bootcamp, and existence of postbootcamp career support. Some bootcamp providers do not limit themselves to mere job search support in the postbootcamp stage. Laboratoria, for example, offers a continuing education option for its graduates and their coworkers comprised of an 18-month educational program of six to eight hours of in-person classes each week. They are currently testing this new program to understand the demand from companies, who would ideally cover half of the program cost, while participants would cover the rest. POSTBOOTCAMP SUPPORT CHECKLIST ⃝⃝ Consider providing job placement services to program graduates, which are not only limited to the search for full-time salaried jobs. ⃝⃝ Build strategic partnerships with the private sector: the more expansive the provider’s network, the higher likelihood that the majority of graduates will be employed after the bootcamp. 27 EVALUATION AND DISSEMINATION OF RESULTS The success of a coding bootcamp should not be measured by the number of press mentions or Facebook likes, or rather not just by them alone. A bootcamp’s success should be determined through rigorous evaluation of how effectively the bootcamp objectives, set at its design stage, have been reached during the timeframe indicated. Ideally, such evaluation needs to rely on a mix of quantitative (for example, surveys) and qualitative research (for example, interviews, focus groups) methods. Otherwise, it may be incomplete or skewed towards certain predefined outcomes. Evaluation targets – or impact indicators – have to be specified ahead of the training: for example, they may have to do with skills attainment (percentage of participants regard their skills as improved in area Y), the level of awareness increased on certain subjects (percentage of participants regard their awareness increased in area Y), contacts made in the ecosystem (percentage of participants made, on average, Y contacts throughout the training, outside their principal prebootcamp network), the number of jobs obtained postbootcamp (percentage of participants obtain Y number of jobs, internships, work offers), and so on. Best practice in project evaluation suggest conducting an intake/baseline survey among all of the bootcamp participants as soon as they have enrolled on the course. A follow-up/exit survey, consisting of the same program-related questions, should be conducted three and/or six months after program for all students. Surveys should measure the impact of the bootcamp on employment as well as the general perception of employability, and so on. It is also useful to conduct an exit survey with those who expressed interest in the bootcamp but did not enroll. Such an exit survey will help measure the differences in employment outcomes between the bootcamp graduates and their comparable peers (that is, those who did not take the training) and thus draw meaningful inferences on the real impact of the bootcamp intervention. To supplement the quantitative data gleaned through surveys, it is highly recommended that either in-depth interviews or focus groups be conducted before and after the bootcamp, targeting randomly selected applicants, participants/graduates, and dropouts across the range of sociodemographic characteristics (for example, age, sex, residency, income, education, and so on). This qualitative data will complement the survey data with new insights, often carrying explanations for certain observed but inexplicable quantitative data inferences. At the same time, some sensitive or difficult aspects of the program implementation (for example, those pertaining to gender dynamics) can be more fully elucidated through qualitative study. To ensure independence of the evaluation, it is recommended that a qualified third party (for example, a data collection consultant or entity) carry out the data collection, data analysis, and produce reports, while coordinating its activities with the bootcamp provider. Participants will be more inclined to share sensitive information and speak up if they are reassured that no retribution may follow. Once the evaluation results have been obtained, a bootcamp provider may want to disseminate the main findings internally, to bootcamp stakeholders, or more widely. These findings could be “packaged” into a comprehensive report, brief fact sheet, a booklet, an interactive presentation or infographics. They could become a “teaser” to drive traffic to the webpage of a bootcamp or its social media outlets. If resources allow, these findings could form the basis of a documentary or promotional video narrative dedicated to the bootcamp results. 28 EVALUATION CHECKLIST ⃝⃝ Set evaluation targets (impact indicators) against the project objectives as early as the project design stage. ⃝⃝ Evaluate the bootcamp through a combination of quantitative and qualitative research methods; outsource this task to a competent and independent third party, as appropriate. ⃝⃝ Consider embedding evaluation results into a variety of communications products and organizing a press event to present them. 29 CONCLUSIONS The skill bias has never been more pronounced than during the Fourth Industrial Revolution. Workers with advanced skills are widely demanded, while those with “regular” skills (read: replaceable by technology) are prone to losing their jobs. Technical skills, such as the ability to develop, maintain, and operate ICT systems, are an example of advanced competitive skills. Unlike common stereotypes, they can be taught in short time spans and applied immediately. One vehicle for that is coding bootcamps. A type of rapid technology skills training, coding bootcamps teach nonroutine tech skills, while also helping students polish their cognitive abilities and interpersonal competencies. Based on the reported employment data from a sample of bootcamp providers, this type of training appears to offer an effective way for rapid reskilling or upskilling of the labor force to meet the current market demand for entry-level IT professionals. As coding bootcamps represent a new phenomenon in the market for education, the question remains whether they can enable their graduates to successfully compete in the labor market over time. To ensure sustainability, they need to be complemented with medium- and long-term initiatives in the areas of early tech education, continuing tech education, and entrepreneurship and with a strong focus on developing socioemotional skills. This toolkit therefore serves as a resource for policymakers who are interested in leveraging coding bootcamps for development impact. Other professionals may also find it useful, especially if they are devising strategies on active labor market policies for young people. 30 NOTES 1 https://jobsanddevelopment.org/. 2 An entry-level technology job is a job that is normally designed or designated for recent graduates of a technological discipline and typically does not require prior experience in the field or profession. 3 Institutions that teaches entry-level employees the basics of a new job, so that they become 100 percent employable from the first day. This reduces the learning curve of new employees within a company, thus saving money and time to the employer. The finishing school can provide training on a particular technology or a particular project. 4 This toolkit has been created with an assumption that any organization, regardless of its status of registration, has a business model, as it strives to viably create, deliver, and capture value. 5 Ruta N Corporation is a public joint venture between the mayor’s office of Medellin, UNE Telco (UNE), and public utility service company Empresas Publicas de Medellin (EPM). 6 The ROI was calculated using data from Course Report 2016a. 31 BIBLIOGRAPHY Course Report. 2016a. 2016 Course Report Alumni Outcomes & Demographics Study. https://www. coursereport.com/reports/2016-coding-bootcamp-job-placement-demographics-report. Course Report. 2016b. 2016 Coding Bootcamp Market Size Study. https://www.coursereport.com/ reports/2016-coding-bootcamp-market-size-research. International Telecommunication Union. 2016. Coding Bootcamps: A Strategy for Youth Employment. http://www.itu.int/en/ITU-D/Digital-Inclusion/Youth-and-Children/Documents/ CodingBootcamps_E.pdf. Meng, Victoria. 2013. “The Ultimate Guide to Coding Bootcamps: The Most Selective Bootcamps,” SkilledUp. http://archive.fo/YCwEA#selection-145.0-145.68. Mulas, Victor, Michael Minges, and Hallie Rocklin Applebaum. 2015. Boosting Tech Innovation Ecosystems in Cities: a Framework for Growth and Sustainability of Urban Tech Innovation Ecosystems. Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/ en/623971467998460024/Boosting-tech-innovation-ecosystems-in-cities-a-framework-for-growth- and-sustainability-of-urban-tech-innovation-ecosystems. Mulas, Victor, Cecilia Maria Paradi-Guilford, Elene Allende Letona, and Zhenia Viatchaninova Dalphond. 2017. Coding Bootcamps: Building Future-Proof Skills Through Rapid Skills Training. Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/ en/795011502799283894/Coding-bootcamps-building-future-proof-skills-through-rapid-skills- training. Ruta N Corporation. Horizontes: Conquering Borders is Our Heritage. http://rutanmedellin.org/ images/descargas/documentos/Revista_Horizontes_ingles_baja.pdf. U.S. Department of Education. 2015. “Fact Sheet: Department of Education Launches the Educational Quality through Innovative Partnerships (EQUIP) Experiment to Provide Low-Income Students with Access to New Models of Education and Training,” October 14. http://www.ed.gov/ news/press-releases/fact-sheet-department-education-launches-educational-quality-through- innovative-partnerships-equip-experiment-provide-low-income-students-access-new-models- education-and-training. The White House. “TechHire Initiative.” https://www.whitehouse.gov/issues/technology/techhire. 32 ACKOWLEDGEMENTS This toolkit was developed by the World Bank in close collaboration with Ruta N (Medellin). The World Bank core team leading the research and technical activities used in this toolkit are Victor Mulas, Cecilia Paradi-Guilford, Elene Allende Letona, Yegana Baghirova, Paola Vargas Gonzalez, and Zhenia Viatchaninova Dalphond. The extended team includes Samhir Vasdev, Domoina Rambeloarison, Hallie Applebaum, Marta Khomyn, Scott Henry and Jerry Zhang, who also contributed to the activity. The team in Ruta N Corporation was led by Ruben Villegas and Ana Maria Ospina. The research and technical activities that led to this toolkit are part of the World Bank’s Jobs Umbrella Trust Fund, which is supported by the U.K. Department for International Development, the Swedish Development Agency (SIDA), the Austrian Development Agency, and the Governments of Austria, Germany, and Norway. The World Bank acknowledges the work of its research and implementation partners for this project, which include Berytech (Lebanon), iHub (Nairobi), and Ruta N (Medellin). Bootcamp providers selected for this activity were SE Factory (Lebanon), Moringa School (Nairobi) and World Tech Makers (Medellin). More information on the World Bank Coding Bootcamps Activity may be found here: http://www. decodingbootcamps.org. 33 CODING BOOTCAMPS: GUIDE FOR PRACTITIONERS This work is available under the Creative Commons Attribution Non-Commercial 3.0 IGO license (CC BY NC 3.0 IGO). Under the Creative Commons Attribution Non-Commercial license, you are free to copy, distribute, transmit, and adapt this work for non-commercial purposes, under the following conditions: Attribution—Please cite this work as follows: World Bank. 2018. Coding Bootcamps: A guide For Practitioners. License—Creative Commons Attribution Non-Commercial CC 3.0 IGO Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Responsibility for the views and opinions expressed in the adaptation rests solely with the author or authors of the adaptation and are not endorsed by The World Bank. Third- party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to the Publishing and Knowledge Division, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@ worldbank.org