DESCRIPTIVE NOTE Results-Based Financing (RBF) for Enhancing Book Distribution in Cambodia World Education, Inc. Submitted to the Results for All Children (REACH) Trust Fund, World Bank Group Contract: 7184295 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table of Contents 1. Executive Summary ................................................................................................................ 3 2. Introduction and Background ................................................................................................ 6 3. Evaluation Approach and Methodology for Track and Trace............................................10 4. Results ...................................................................................................................................12 A. Access................................................................................................................................. 12 B. Distribution System Performance........................................................................................ 15 i. Requests ......................................................................................................................... 15 ii. Disbursements ................................................................................................................ 16 iii. Time Taken to Deliver ..................................................................................................... 18 iv. Successful Deliveries ....................................................................................................... 19 v. Overall Feedback About Distribution System Performance.............................................. 21 C. Community Engagement..................................................................................................... 22 D. Results-Based Financing (RBF) Efficacy ................................................................................ 24 5. Next Steps for Track and Trace in Cambodia .....................................................................27 6. Conclusions and Policy Implications for Track and Trace in Cambodia ............................30 7. Analysis and Recommendation for Design and Pricing of TnT in Other Contexts ...........35 Annex 1: Track and Trace Evaluation Plan ..................................................................................38 Annex 2: Illustrative Case Studies ................................................................................................40 Annex 3: Template for Recording Book Request Data ...............................................................44 Annex 5: Data Tables ....................................................................................................................47 Table 1: Percentage Who Responded That All Students Have All Textbooks (by school category) (NEP) ........................................................................................................................................... 47 Table 2: % of Teachers and Students Who Said Students Received Textbooks (by grade and subject at baseline and endline) (NEP) ..................................................................................................... 48 Table 3: Responses to the Question: “Did students receive textbooks from school on time (during November)?” (NEP) ..................................................................................................................... 49 Table 4: Responses Regarding Which Months Books Were Received Among Those Who Stated That Books Were Not Received On Time (NEP) ............................................................................ 50 Table 5: Percent Received vs Approved (Salesforce) .................................................................... 51 Table 6: Spot Check Results (Salesforce) ...................................................................................... 52 Table 7: % of Books Received Compared to Books Approved (by grade and subject).................... 54 2 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 1. Executive Summary Ensuring that educational materials are delivered in the correct quantities to the schools they are intended for, and in a timely fashion, is a challenge in many countries. In Cambodia, research conducted by World Education (WEI) and John Snow, Inc. (JSI) highlighted a well-known picture of a textbook supply chain that is characterized by delay, inefficiency and inaccuracy at every stage. The result is that new textbooks do not arrive in time for the new school year, with up to three other deliveries arriving considerably later, and deliveries rarely match requirements. In addition to impacting the quality of education, the loss of inventory and mismanagement of resources results in waste and inefficient spending. Materials can go astray at any stage in the delivery process, but without the availability of tracking information, officials do not know where the issues arise. To address this challenge, WEI and JSI designed Track and Trace (TnT), a technology solution that is designed to increase the visibility and efficiency of the textbook supply chain. WEI and JSI’s Enhancing Book Distribution in Cambodia project funded by the World Bank under the Reach for Reading initiative, set out to address a number of the supply chain weaknesses in Cambodia, while acknowledging that it cannot address them all. Working in close conjunction with Cambodia’s Ministry of Education, Youth and Sport (MoEYS), the initiative aimed to demonstrate TnT’s ability to make the primary school textbook supply process more efficient by three main features: 1. Schools report timely data and TnT assists in accurately projecting book needs for budgeting and procurement by the central ministry 2. Tracking “real-time” distribution of textbooks and providing visibility of issues – delayed, missing or damaged – for taking timely action 3. Engaging school communities to confirm books arrive and conducting classroom spot checks post-delivery to ensure textbooks are available It has shown success along each of these. Overall, 100% of the 416 pilot schools across 10 districts successfully used the Track and Trace digital platform on their phones to complete book requests within 2 weeks. That data was then made immediately available to the central government (compared to 6 months under the usual paper-based process). During book distribution, all pilot schools and District Offices of Education submitted digital receipts and dispatch records on TnT. TnT generated comprehensive data on the progress and accuracy of deliveries. This was available to MoEYS officials from district up to central level through web-based dashboards and reports. For instance, TnT was able to inform key officials in real time that 86% of deliveries to schools were received in January, 5% in May and 9% in June. It also showed that while 99% of the total number of books approved by MoEYS were received by school directors in pilot schools, 94% of school directors’ digital receipts matched exactly the approved quantities of textbooks allocated to that school. School Support Committees from the majority of pilot schools engaged with Track and Trace. 50% of deliveries were confirmed by the SSCs for the first round of distribution in January and 75% for the 3 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 second round in May / June. Additionally, 92% of schools had an SSC that conducted at least one spot check of book availability at their school. There have been encouraging indications regarding the Results Based Financing aspect of the pilot. Leaders from within central MoEYS, including the Procurement Review Committee, have commented that in the future, once TnT is used at scale and demonstrated the reliability of its data, TnT data could potentially be used as an additional criterion for releasing the 10% deposit to distributors. It can be concluded that in this pilot the TnT platform has demonstrated to MoEYS its ability to generate the data needed for performance-based contracting of printers/distributors. There is now a growing recognition within MoEYS that TnT could be used for this purpose once TnT is applied at national scale. The project also tested out incentivizing School Support Committees (SSCs) to be involved in textbook supply by asking District Offices of Education (DOEs) to award certificates to SSCs who conducted all of their expected tasks on TnT. In total, 60% of SSC’s fulfilled the criteria and were eligible to receive a certificate. All ten DOEs printed out certificates for these SSCs and presented them during school opening ceremonies. Endline focus groups revealed support for the continued use of certificates as an incentive, although findings also suggest that the opportunity to have a practical role in monitoring textbook supply is sufficient motivation for many SSC members. Overall, TnT has demonstrated its potential to bring improvements to many of the weak spots in the supply chain, although the system is not be able to address challenges with procurement processes that are outside the project’s influence. TnT has provided MoEYS with comprehensive data that makes clearly visible the current poor performance of the book supply chain in Cambodia. Feedback from users at all levels and MoEYS senior leaders has been extremely positive, and all have indicated a desire for TnT to continue to be used. The Minister of Education declared in May 2019 that MoEYS will Cambodia’s Minister of Education with members of World Education Cambodia and take over TnT and scale it up World Education’s EdTech Center nationally from 2020 and the Publishing and Distribution House under MoEYS has committed to paying for TnT’s running costs. There have also been promising indications that the MoEYS Procurement Review Committee will consider using TnT data to help determine final deposit release to distributors once TnT is used at scale. In this way, MoEYS would be using TnT data to hold printers/distributors accountable for 4 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 delivering the right books to the right schools at the right time and in the right condition - the project’s ultimate goal. Challenges Identified at Design Solution Achievements Baseline School Directors were TnT calculates book TnT calculates all book projecting book needs two projections reducing projections automatically years in advance using a calculation errors complicated formula It takes the MoEYS six months MoEYS have book requests 100% book request data were to enter all requests in Excel immediately received the two week spreadsheets reporting period MoEYS manages all book MoEYS were still required to This will be addressed in phase projections, budgets and download requests from TnT 2 distribution lists in Excel and manipulate based on spreadsheets budget Procurement process takes 2 As MoEYS have book requests This could not be implemented years from time SDs complete immediately they can use for during the pilot as the request to when they receive current year distribution procurement process for the the books year was already in process PDH manually enters PDH is able to access the This functionality existed distribution list for each school distribution list from TnT and however as the pilot was only into a MS Access system from print receipts for 400 schools it did not make hard copies provided by sense for PDH to use TnT at MoEYS this stage There is no way to track if DOEs, schools and SSC+ send 99.35% of SD’s sent receipts shipments reach all schools in receipts that books have on receiving the shipment arrived 44% and 70% of SSC+ confirmed receipts during round 1 and 2 Receipts take weeks or months SDs and SSC+ confirm the 94% of SD’s digital receipts to be collected quantities of books received matched the approved quantities of textbooks allocated to that school 94% of SSC+’s receipts matched the quantities received reported by SDs To check if books are available Revitalize SSCs giving them a 92% of SSC+ conducted in classrooms MoEYS must role in text book distribution classroom spot checks visit schools 5 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 2. Introduction and Background Background and Context: Since July 2017, World Education, Inc. (WEI) has received funding from the World Bank under the Reach for Reading initiative to implement the Enhancing Book Distribution in Cambodia project. Under this project, WEI, its sister organization John Snow Inc. (JSI) and the government redesigned and piloted the Track and Trace digital platform. Track and Trace (TnT), originally developed by JSI and WEI with funding from the All Children Reading Grand Challenge, is a technology solution that enables ministry officials, partners and distributors to track in real time the ordering and distribution of books to ensure that they reach the schools they were destined for in a timely manner and in the right condition. TnT improves the transparency and accountability of the education supply chain by enhancing the central government and community’s visibility into the location of textbooks during the distribution process. Early grade classrooms in Cambodia rarely have sufficient supplies of textbooks to meet the needs of all children. Findings from baseline, end-line, ongoing field monitoring and ministry consultations during the implementation of the first phase of the Enhancing Book Distribution in Cambodia project have all confirmed a well-known picture of a textbook supply chain that is characterized by delay, inefficiency and inaccuracy at every stage. The result is that across the country, new textbooks do not arrive in time for the new school year, with second, third and fourth deliveries arriving considerably later, and contents of deliveries do not match requirements. Under the first phase of the Enhancing Book Distribution in Cambodia project, WEI, JSI and the Ministry of Education, Youth and Sport (MoEYS) designed the Track and Trace platform to address a number of the supply chain weaknesses, while acknowledging that it cannot address them all1. The MoEYS- and WEI-agreed upon objectives of the project were to help MoEYS improve the textbook supply chain by providing timely data and accurate projections of book needs for budgeting and procurement, The TnT digital platform uses cost-efficient tech tracking “real time” distribution of textbooks and providing tools including a chatbot, a smartphone app, visibility of issues. The project also aimed to engage the and a web-based dashboard school community to ensure textbooks are available for the students and it was envisaged that data from Track and Trace (TnT) could be used in the future to inform results-based financing of distributors. Synopsis of Project Activities: The Enhancing Book Distribution in Cambodia project, now approaching the end of its first contract with the World Bank, has achieved a number of successes. The TnT platform was developed between July 2017 – April 2018 through very close senior MoEYS involvement and field research. It was ready for piloting in time for book requests in May 2018. Government engagement was a fundamental part of the pilot and was established right from the proposal stage in late 2016 as senior staff from the Primary Education Department helped to lead the proposal development process. The Minister of Education, HE Dr. Hang Chuon Naron, further confirmed engagement with the official project endorsement in April 2017. In early September 2017, there was the formal nomination by Deputy Minister, H. E. Nath Bunroeun, of the National Working 1 See Annex 4 for a summary of baseline findings 6 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Group for the Enhancing Book Distribution Project with nominated officials from relevant departments, chaired by H.E. Puth Samith, Director General of the General Directorate of Education. Departments that were represented included the Primary Education Department, Department of Curriculum Development, Procurement Unit, Department of IT, and the Publishing and Distribution House. MoEYS were involved in the field level baseline and feasibility assessment and there were intensive design consultations throughout TnT’s development, with a remarkable level of engagement by the top level of MoEYS leadership. The proposed, final TnT design was presented by the National Working Group and WEI to the Secretary of State for Education on December 8, 2017 for final review. Approval granted to proceed with building and implementing the Track and Trace system. For the remaining duration of the project, MoEYS counterparts have been involved with training package design, training facilitation, and field monitoring, in addition to regular progress meetings and consultations with the entire National Consultative Group at critical pilot junctures. The final TnT platform utilized various communication technologies to enable reporting of information and tracking of books throughout the ordering and distribution process. As can be seen in the graphic below, the technologies included barcodes, smartphone applications, chat-bots, and web-based dashboards. The digital information management system created by WEI and JSI has enabled MoEYS and stakeholders to trace books through the various ordering and distribution points, and to track quantities received compared to quantities ordered. The project also tested ways to incentivize the school community to send messages through TnT, respond to messages, and play a role in ensuring books arrive to and are used routinely in school classrooms. 7 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 The TnT platform went ‘live’ in May 2018. Overall, all 416 pilot schools across ten districts in two provinces (Banteay Meanchey and Prey Veng) successfully used the chatbot function of TnT on their phones in May to complete book requests within two weeks. That data was immediately available to the central government (compared to six months under the usual paper-based process). In January 2019, for the first book deliveries, staff at the Publishing and Distribution House (PDH) used an app on TnT to scan barcodes of book shipments for the 416 pilot schools as they left the warehouse. Once the books arrived at district level, the barcode was scanned again to confirm their receipt. Arrival of the books at all 10 District Offices of Education (DOEs) were then tracked by government officers using barcodes and an app on mobile phones, and all 416 school directors used the chatbot function to record exact numbers and titles they received. School support committees (SSCs) at schools were notified that books had arrived at their school and then used TnT to record the number and title of books unpacked. This process was repeated for the second round of book deliveries carried out by the PDH in May/June 2019, and a modified process was used for the deliveries conducted by a second contractor2. After final deliveries, the SSCs then used TnT to conduct book spot checks in classrooms. The pilot has been such a success that the Minister of MoEYS stated on April 25, 2019 that he wants TnT to be used at scale in Cambodia, and that MoEYS will pay associated license/running costs (approximately $13,000 per year at national scale at current estimates). It was later agreed that funds will come from the Publishing and Distribution House (which is a state-owned enterprise under MoEYS), at the offer of the Director of the Publishing and Distribution House, and not from regular MoEYS funding streams. This will make payment for the running costs of TnT much more straightforward. In order to ensure a smooth rollout in 2020, the Minister has asked that WEI pay for the required modifications to the platform and a limited amount of trainer training. WEI is seeking the funding needed to perform these tasks. Results-Based Financing Components: The pilot aimed to explore results-based financing and incentives to improve performance at the top and bottom of the textbook supply chain. At the top end of the supply chain, the intention was that TnT could provide data for monitoring the performance of the distribution contractor/s. The project learned fairly early on that 100% of the contract value is paid by the government to the contracted printer/distributor before delivery, as government funds must be spent by the end of the calendar year. However, the contractor has to pay 10% of the value of the contract into a bank account as a deposit. This deposit is released after checks of book delivery receipts have occurred by the government’s Procurement Review Committee. According to the Procurement Unit, the project team learned that the deposit can only be released when any found issues have been resolved. As a result, the project identified the 10% deposit release as the only RBF entry point for TnT. The long- term goal is that TnT data will be able to assist the Procurement Review Committee to approve the release of the final 10% deposit for the contractor. At the bottom end of the supply chain, the project targeted School Support Committees. This decision was made in light of widespread unanimity during the baseline/feasibility research regarding SSC involvement in textbook monitoring and revitalization of SSCs in general. It was decided that SSCs would receive formal recognition if they fulfilled the following criteria:  At least one registered SSC member on the TnT system 2 At time of TnT design, there was no indication that more than one contractor would be used for book deliveries. This caused significant difficulties as the system did not allow for multiple deliveries, and furthermore, MoEYS did not enable WEI to communicate/coordinate with the second contractor regarding the use of TnT. The project team found ways to work around these obstacles and ensure that data on book distribution by the second contractor did end up being recorded in TnT. 8 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295  Submit book numbers received at school  Conduct classroom spot checks and report information It was agreed by the ten pilot DOEs that they would use TnT data to determine which schools have SSCs that would receive the recognition. They would then use their own budgets to recognize the SSCs through providing a plaque or certificate to be displayed prominently at the school. The report that follows provides a detailed examination of the outcomes of the project. Firstly, the methods used during the project evaluation are explained. This is followed by a presentation of the results and a concluding section that offers a summary of the outcomes, conclusions, and recommendations. Implications for further application of Track and Trace in Cambodia and in other contexts are examined. 9 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 3. Evaluation Approach and Methodology for Track and Trace In order to assess the success of TnT to date, four dimensions were measured. Access Measurements: Data regarding the % of sampled students without books/schools with book shortages and other indicators measuring overall access to textbooks were collected in a number of ways. 1. Survey using a modified, digital version of the Textbook Policy Checklist monitoring tool that was originally developed by the NGO Education Partnership with funding from the EU. This tool was used in a sample of 42 of the 416 pilot schools at baseline in August 2017 and was reapplied after book distribution in August 2019. 2. Spot checks were carried out at the end of June 2019 with a sample of twenty schools (10 in each province) to verify that the newly arrived books had actually reached and were being used by students. 3. Post-distribution surveys implemented by School Support Committee members (SSCs) through the TnT platform via chatbot to track whether students have textbooks. Distribution System Measurements: To measure the efficacy of the distribution system in pilot districts and identify weak spots the project used data from TnT. TnT allowed for tracking indicators within the project’s pilot area such as: Requests: ● % schools submitted book requests within approved timeframe Disbursements: ● % of any district shipments received at DOEs ● % of school directors that received all shipments Time taken to deliver products ● % of deliveries to schools received by month Successful deliveries (right contents and right condition) ● % school directors’ receipts that match approved quantities in allocation list ● % of total books received by SDs compared to approved quantities in allocation list by grade and subject ● % of books confirmed as received by SCC+3 members compared to quantities received by SDs Additionally, the project administered a number of focus group discussions and interviews at the end of the project to collect more qualitative data regarding experiences under the new TnT system. These were conducted with teachers, School Support Committee members and school directors in the 42 schools where the baseline was conducted in August 2017, and with involved DOE and POE officials. Three different focus group discussion (FGD) tools for school directors/librarians, School Support Committees (SSCs), and District Office of Education (DOE) officials were made. The endline FGDs were held in six of the ten target districts across Banteay Meanchey and Prey Veng. In each district, for the school director/librarian FGDs, one school director/librarian was invited from each of the primary schools in the previously selected sample for administration of the Textbook Policy Checklist (described above under Access Measurements). Additional school directors/librarians from other target schools were invited so that each FGD had between 7-8 school directors/librarians. A similar approach was used for the School Support Committee FGDs in each district. An FGD was also held in each provincial town to which the DOE officials responsible for textbook supply in each 3 The project uses the term SSC+ to denote the fact that SSCs have been encouraged to informally recruit an extra member - an assistant - who can help them with the use of digital technology if needed. 10 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 district was invited for discussion. Lastly, interviews were held in each province with the personnel from the POE responsible for textbook distribution. In total, 77 school directors/librarians, 35 SSCs, 9 DOE officials and 1 POE official took part in the endline FGDs and interviews. Furthermore, during the book request period the project team carried out visits to a sample of schools and used a monitoring form to interview school directors about using TnT. This monitoring form was administered to 87 school directors, equivalent to 21% of the total number of pilot schools. The project also created a monitoring form for book distribution to investigate experiences by DOEs, school directors, and SSC members of using TnT for delivery. The tool was used with 10 DOE officials in charge of textbook supply, 130 school directors, and 76 SSC members across Banteay Meanchey and Prey Veng provinces. Lastly, illustrative case studies (written and video) were compiled with 4 school directors at various times during the textbook supply process (requesting, disbursement) to gather an in-depth picture of using TnT throughout the textbook supply process from the end-users’ point of view. Community Engagement Measures: In order to understand the level of engagement by the school community, the project tracked School Support Committee (SSC) interaction with the system by using data from the TnT platform. Specifically, TnT generated data for the following indicators: · % schools with at least one SSC member registered on TnT · % SSCs that have counted the books received by their school · % SSC that have conducted at least one spot check of book availability Again, qualitative data from interviews and focus groups with SSCs, teachers and school directors were used to augment the quantitative data from TnT. Results-Based Financing (RBF) Efficacy: Findings from focus groups with SSCs, DOEs and school directors generated qualitative data on the perception and receptiveness of SSCs to the community-level RBF element of the project. Interviews with the central ministry gathered information about the second RBF element, the efficacy of TnT in determining the release of the deposit to the publishing and distribution contractor/s. Five MoEYS officials were interviewed, all of whom were familiar with TnT through their involvement in the National Consultative Group. The officials interviewed included: 1) H.E Chan Sohea, PED director; 2) Ouk Thoeurn, Deputy Director General, PDH; 3) Sok Tha, DIT Director; 4) Lim Sotharith, Chief Office, DCD; and, 5) Ren Sopheap, Chief Office, Procurement Unit. In addition to these interviews, the project has collected opinions and statements from senior MoEYS leadership regarding TnT’s future role in holding printer/distributor contractors accountable over the course of the pilot. Annex 1 provides a summary of the evaluation activities conducted by the project to assess success. 11 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 4. Results The evaluation measured project outcomes along four dimensions: access; distribution system performance; community engagement; and, RBF efficacy. Results under each of these dimensions are discussed in turn below. A. Access Various approaches were used to investigate whether students had books and/or schools had book shortages. These include a survey conducted in 42 schools, spot checks by the project team and classroom visits by School Support Committees. Survey: The Textbook Policy Checklist monitoring tool was used in a purposive sample of 42 schools at baseline in August 2017 and was then reapplied after book distribution in August 2019. This survey investigated book access in general, and was not specifically about the newly delivered books. Results from this survey, comparing baseline to endline, do not show a clear pattern and suggest that ensuring access is multifaceted. The following graph is disaggregated by the distance category, defined as: Category 1: urban (DOE is 19km and less from the POE); Category 2: semi-rural (DOE is 20km-40km from POE); Category 3: rural (41km and over). Overall teachers and school directors reported a reduction in the % of students who had all textbooks when comparing baseline to endline, especially in category 1, the schools closest to DOE and POE. When students and teachers were asked if students had been provided with textbooks for the following subjects and grades we saw mixed results when comparing baseline to endline. Students reported an increase in the number of students being provided with English books, but this does not correspond with teachers’ reports. This demonstrates the difficulties associated with monitoring the general availability of books with surveys. It should be noted that respondents were not asked to comment on whether the books they had been provided with were old or new, or usable. 12 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Project Team Spot Checks: These were carried out by the project team with a random sample of schools to verify that the newly arrived books had actually reached and were being used by students in the classroom. In total, 20 schools across ten target districts in the target provinces were visited in the second half of June. The school directors were surveyed regarding the new textbooks, and classrooms were visited to observe whether children were using the new books. It was found that the books from the first book SSC members conduct a classroom spot check at a school in Prey Veng province deliveries in January had been distributed to 13 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 students and there was evidence that students were using them in the classroom. However, regarding the last book deliveries of the year that occurred at the end of May and first week of June, it was found that only 55% (11) of school directors said that the more recently delivered book titles (English Grade 4, English Grade 6, Khmer Grade 1, Khmer Grade 3) had been distributed to students for use. The TnT team went into classrooms to verify this, and found that students at these 55% of schools were indeed using the newly arrived book titles. Among the 45% (9) of school directors who reported that they had not distributed those recently delivered books to students, the reasons provided were that it was nearly the school vacation and/or that school directors had given up waiting for the new copies of the books and had decided to use and distribute the old books to students. The project team verified that the versions of these book titles they observed students using were generally very old, damaged books and often without a cover. This is a significant finding and demonstrates clearly the effect of such late deliveries. SSC Post-Distribution Surveys: These were implemented by SSC members through the TnT platform via chatbot after book distribution to track whether students have textbooks. Unlike the project team spot checks, these were not investigating whether the newly arrived books were being used specifically, but whether students had books in general. Data from TnT shows us that 92% of SSCs conducted at least one spot check of book availability at their school. All spot checks were conducted between June and August after all deliveries were complete. For all spot checks across all grades and subjects, at least 80% of students had a textbook. Conclusions about Book Access: Overall, it can be summarized that book access showed no improvement. These poor results concerning access are not a negative reflection on Track and Trace, however. Over the short term, it is not to be expected that TnT would be able to impact book access, given that TnT has no influence over procurement policies and contracting. The 2018-2019 school year has been fraught with procurement issues between MoEYS, the PDH and an Independent Publishing House, with the final contract only being signed at the end of December 2018/early 2019. As a result, printing of some book titles was still occurring well into 2019, and deliveries were taking place as late as May and June 2019, two months before the end of the school year. Thus, for the majority of the year, students did not have all their book titles. 14 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 B. Distribution System Performance i. Requests Based on data generated by the Track and Trace platform (using Salesforce), 100% of schools in the 10 pilot districts submitted book requests in May using TnT within the approved timeframe of two weeks. This data was then available for immediate use by staff from the Department of Curriculum Development (DCD) at central level. This compares very favorably with the usual six-month period that it takes for request data to reach the central ministry from all schools, as reported during the baseline. Overall, findings from the Focus Group Discussions suggested that the majority of school directors and librarians in participating schools were pleased with the requesting function of TnT and found using TnT for book requests to be easy. The main reason given was that the requests were done close after the training or even during the training provided by WEI and partners. Respondents stated that they had no problems using the Telegram chatbot and the structure of the forms were easy to fill in. Among the few issues that were raised by a small number of participants were Internet connectivity, low digital skills, and difficulties finding the TnT request link. These issues were echoed in the responses given by school directors during school monitoring visits by the project team. These visits were made to 87 school directors whom project staff had identified as potentially needing extra support. Among this “at risk” category, only 13 school directors (or 15%) needed actual assistance in locating the request form on the system before being able to fill in the data themselves. The four illustrative case studies of school directors conducted by the project team illustrate similarly positive opinions about using TnT for book requests, as expressed in general by the FGD participants. For instance, the school director at Kleng Por primary school in Svay Check district, Banteay Meachey province explained that using the TnT system is far easier for him, because he doesn’t need to spend much time calculating the book request by himself. He stressed, “Doing the book request via TnT system is useful, because it is faster, easier, and I can see the calculated results in Telegram message, and all levels can also see the Mr. Chanthol, a school director in Banteay Meanchey province, requests textbooks using book request at the same time.” Another case study pointed the TnT system out that “submitting book requests through the TnT system is faster and easier than requesting the textbook by paper, because it saves time and it is easy to correct the data when we made mistakes.” “Filling the book request using paper is more complicated than the TnT system since TnT system calculates the request amount automatically, ” said another school director from Prey Veng. Case study subjects commented positively on the TnT feature that allows them to re-enter their request data if they made an error, even after initial submission. “TnT system is very useful to receive the calculated data after the book request and correct and resend it if the data does not match,” was one such positive comment from a school director in Banteay Meanchey. The same school director declared that the paper method takes too long. Qualitative data from officials from the District Offices of Education (DOEs) has also been positive regarding the use of TnT for requests. DOE participants in the FGDs across target provinces reported that TnT was useful and important in terms of their role in overseeing textbook requests. According to FGD comments from DOE officials in charge of book requests, TnT reduced some of their tasks because they no longer need to compile and combine school book requests for their districts, and transport the lists to the POE. He raised the point that using the paper method to make book requests can cause errors because sometimes the DOE may unintentionally type the wrong number 15 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 or incorrectly calculate the book request, which causes errors to the amount his school should receive. Finally, as central level users of the system, the MoEYS officials from relevant departments who had been given access to TnT were asked whether they utilized the dashboards and report functions of TnT during the requesting or disbursement period. Overall, among the five people interviewed, very few visited the dashboard more than a couple of times. The only department to use the dashboard was the Department for Curriculum Development. This is natural given that they were responsible for collecting textbook requests and ensuring correct disbursement nationally. The other department/agency that might have been expected to have accessed the dashboard was the PDH. However, the official from the PDH who was a member of the National Consultative Group for the project stated that she did not use the dashboard at all, as she needed more support and guidance about how to log in to the system. It should be noted that going forward, the Director of the PDH has committed to placing staff with higher digital skills in charge of TnT. The Office Chief from the Procurement Unit echoed similar sentiments from the other MoEYS officials in his response. He did not use the dashboard, as this was only a pilot. All commented that once TnT was scaled up, they would use the dashboard more frequently. In conclusion, it can be said that the response to the book request functions of TnT were extremely positive. Ultimately, its efficacy was most clearly demonstrated in that all 416 schools in the pilot were able to enter their book requests using TnT within just two weeks, with some getting additional, virtual or in-person support from DOE or WEI staff. FGD respondents stated that they did not want to stop using the TnT platform and go back to paper-based requests. ii. Disbursements Data from TnT shows that 100% of district shipments that were dispatched from central warehouses were received at DOEs and 99.35% of school directors received all disbursed shipments (each school director received multiple shipments). While all the target schools in the pilot used TnT to record book distribution, this was not without difficulties. During the FGDs with school directors and librarians, a number of negative comments were gathered about the experience of using TnT during book delivery. School directors/librarians reported difficulties using the system, primarily due to the long gap between the training and book delivery. Indeed, training had been provided in April/May 2018 and book delivery did not occur until January - June 2019 due to Textbooks are delivered to a DOE in Prey Veng province extensive delays in procurement processes. FGD participants also reported confusion created by having multiple book deliveries. This resulted in different links being sent for each new disbursement. School directors became confused over which links they should use. This problem with the system occurred because WEI had been informed in 2017/early 2018, during the design of TnT, that there would only be one distributor company. It turned out, unfortunately, that there were two distributor companies and both made multiple deliveries. The system, as designed, could only track one disbursement at a time from packing through to confirmation at schools, with the confirmation link sent to each contact ID (school director, DOE or SSC) matching a specific book dispatch ID. The system could not accommodate two distributors distributing simultaneously without creating different links, and this caused significant confusion among users. As noted, the project team had not anticipated this situation, and had to work hard to find on-the-ground solutions to this during deliveries. School directors experienced 16 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 other issues related to having to reactivate their account with the Telegram messaging app (if the account is not used in 6 months it is automatically closed) due to the unexpected delay between requests and disbursements (due again to the procurement issues) , as well as low digital literacy skills. Such findings have already proved most useful for informing TnT design and training approaches in the next phase in Cambodia (see later sections on Next Steps, Conclusions and Implications). Findings from monitoring visits and interviews with 130 School Directors and 76 SSC members during delivery confirm these findings above. Of those who were interviewed, 24% of SDs and 40% of SSC members could not confirm textbook receipts on TnT without some additional guidance from DOEs/TnT trainers. Those reasons included limited knowledge of technology, not remembering how to use the system properly due to the training being a long time ago and needing to re- register/install the system. Nevertheless, after receiving guidance, the majority were able to proceed with using the system by themselves. DOE officials from all 10 DOEs were also questioned during monitoring visits. Regarding using the system to record receipt of textbooks from the PDH and to record allocation of books to schools, all DOE officials in charge of textbook supply in target provinces reported they did not face any challenges or problems. DOE participants were also queried during FGDs about their experiences utilizing the TnT system. All DOEs in both FGD groups reported that they did face difficulties logging in to the system to view the dashboard for book requests and book receipts due to their limited ICT knowledge, lack of familiarity with the system and poor Internet connection. Furthermore, one DOE FGD group in Banteay Meanchey said a challenge of distributing the textbooks to schools using TnT was that it took them a lot of time to allocate and distribute the textbooks. We were very pleased to note that all 130 SDs interviewed during school monitoring visits reported that when they came to collect the textbook from the DOE or school cluster, they unpacked the textbook boxes and counted the number of books and damaged books before they signed on the book receipt. Our baseline study had indicated that normally School Directors sign the paper receipts without counting and checking individual books when they collect their new books. This gives hope that TnT is stimulating better practices in book distribution that promote accountability. Positive comments were A school director uses the TnT platform to frequently found during endline regarding this confirm that he received the correct number of textbooks requirement of TnT to count and record actual books received, rather than to simply confirm that the amount specified on the paper receipts were correct. One case study school director noted, “Distributing books through TnT system is more accurate and accountable in terms of filling in only the actual number of textbooks received without depending on the number of textbooks in the receipt.” Very similar feedback was received by the second case study from Banteay Meanchey. “Using TnT system for textbook distribution is more accurate and accountable because filled in TnT system based on the number of actual textbooks received, not the number of textbooks in receipt,” the school director stated. Overall, each of the four case studies had positive things to say about using TnT during distribution. “Confirming the book received through TnT system is faster, easier, more effective and useful, and time-saving”, commented the school director from O Sampor II Primary School in Malai district, Banteay Meanchey. 17 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 A common theme across all FGDs was a suggestion to provide more training on how to use TnT. This will need to be taken into account for the scale up of TnT. School directors also requested adding a feature to TnT that would allow them to generate a report of their book request and book receipts that they could use as hard copy evidence. iii. Time Taken to Deliver For reasons beyond the influence of this project, book deliveries were extremely late this year. As noted already, this was due to procurement issues and conflicts between distributors and MoEYS over contract terms. TnT data shows that 86% of deliveries to schools were received in January (almost three months after the start of school), 5% in May and 9% in June. General dissatisfaction about this late delivery was detected across the FGDs. All FGD groups requested that in the future MoEYS provide textbooks before school starts. NEP data showed an increase in school directors, students and teachers reported that they did not receive books on time (see annex). Percentage who responded that books were received on time (NEP) Baseline Endline Student Groups 89% 70% Teachers 91% 54% School Director 55% 13% % of SDs reporting which months books were received among those who stated that books were not received on time (NEP) 60% 50% 40% 30% 20% 10% 0% December January February March April or Later Baseline Endline Despite the late delivery, it is interesting to note that some FGD participants commented that TnT assisted in speeding up delivery once books had left the warehouses. FGDs with DOE officials in charge of book requests found that TnT resulted in quicker distribution to schools - generally a 1-2 day distribution. During the FGDs, all DOE members mentioned that, under the previous system, when the textbooks arrived at the DOE, they waited approximately 10 days for all school directors to collect their textbooks. One FGD group in Banteay Meanchey reported that this, in turn, saved money they usually had to spend on hiring labor to help carry and store the textbooks inside DOE venues. 18 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 iv. Successful Deliveries Successful deliveries means that the right books are delivered in the right number. TnT has successfully been able to provide data regarding the accuracy of deliveries, fulfilling the project’s objective of creating a digital system to ‘shine a light’ on the textbook supply chain for increased visibility and accountability. Presented below are the types of data that the TnT system has been able to generate which allow various stakeholders to track textbook supply performance. Prior to the deliveries taking place, the final allocation list as per the final contract for the printing house was uploaded into TnT. This enabled comparison within TnT of the quantities of books received as reported by SDs and SSCs with the approved quantities per school as outlined in the allocation list. This approved quantity should also have matched the paper receipts printed by the printing house prior to distribution and received by the schools. The printing houses did not, however, print receipts directly from TnT so this cannot be confirmed. Data from TnT shows that 99% of the total number of Textbooks are delivered to a school in Banteay Meanchey province books approved by MoEYS were received by school directors (see table in annex). However, in pilot schools, 94% of school directors’ digital receipts submitted to TnT matched exactly the approved quantities of textbooks allocated to that school. The fact that almost 100% of the total number of books approved were delivered but the number of receipts with discrepancies is higher can be explained by the fact that discrepancies included both oversupply and undersupply. Of the 6% of receipts with discrepancies, 61% had missing data, 14% were oversupplied and 24% were undersupplied (one receipt could have multiple errors). This demonstrates that while overall the printing houses delivered the correct total number of books, these books were not always allocated according to the paper receipts to the schools. This could have been an issue at the district level where books are allocated to schools. The following graph breaks the percentage of total books received by subject and grade. The graph shows some variation by subject and grade, however it should be noted that graph begins at 90%, as the discrepancy is less than 2% for all books. 19 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 *N.B. note that chart begins at 90%, however it does demonstrate variations across subjects SSC members were asked to digitally confirm receipt of books using TnT when the books arrived at the school. During the first delivery in January, only 44% of SSC members submitted a confirmation of receipt through TnT, partially due to a temporary fault within the system that resulted in SSCs not getting a reminder notification. By the second round of deliveries in May/June, 70% of SSC members submitted a digital confirmation of receipt. This increase in the SSC receipts was due to an increase in Members of the SSC use TnT to digitally confirm the support to the SSC and reminders. receipt of textbooks Of those digital receipts submitted by SSC members, 97% matched the digital receipt submitted by the school directors through TnT for the same school. The discrepancies ranged from 200% over (i.e. 20 received by SSC compared to 10 received by the school director) to 50% under the amount reported by the school director. These large discrepancies were not common and most were only a few books different. The reasons for the discrepancies are unclear; it could be a data entry error on the part of the school director or the SSC, or it could be due to books going missing between the DOE (where the school director counted and picked up books and submitted digital book receipts) and the school (where the SSCs counted and submitted digital book receipts). However, for about 50% of the discrepancies, the SSC reported the quantity received as greater than what was reported by the SD, suggesting an error in the counting of the books. Summary Table of Receipt Data Accurate Receipts Data Results % of SDs receipts matched approved quantities 94% % of SSCs receipts matched SD’s receipts 97% Discrepancies Data for SD receipts* % of discrepancies reported by SDs as oversupplied 24% 20 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 % of discrepancies reported by SDs as undersupplied 16% % of discrepancies due to missing data in receipt submitted by 61% SDs *% out of total discrepancies In addition to the data generated through TnT regarding textbook delivery accuracy, the project team also collected data during field monitoring and endline evaluation activities. During spot checks by the team to 20 schools to check whether the newly delivered books had been distributed to students, school directors were also asked about the accuracy of the book deliveries. 100% of school directors surveyed during these spot checks said that the books received matched perfectly with the amounts on their paper receipts (i.e., with the approved allocation list). Similarly, questions were asked during FGDs with different stakeholders to ascertain their perceptions of the accuracy of book distribution this year. All FGD groups (school directors/librarians, DOEs, SSCs) across the target provinces stated that their schools received textbooks matching with the quantities and subjects on the paper receipts provided. Paper receipts are provided by PDH and the second Independent Printing House with the deliveries based on approved quantities in the allocation list in contract. Beyond this, other FGD questions explored if the textbooks received matched with the schools’ real needs. Interestingly, the results found that in three of six FGD groups with school directors/librarians, participants stated that the received books did not match with the school needed. In addition, during the FGD with the five DOEs in Banteay Meanchey, it was estimated that 20% of schools reported to DOEs that the books received did not meet the school needs. These findings do not necessarily have any bearing on the accuracy of distribution as per the paper receipts/allocation list, but do indicate clearly the problems inherent in asking schools to request books two years in advance. As seen here, the final approved allocation list may not align with needs two years later. Fortunately, the new design of TnT will help to eliminate this issue as the PDH will now be able to use request data generated speedily through TnT for the same year’s book distribution. In general, the picture provided by TnT data, endline evaluation activities and monitoring reports is one of a book distribution in the pilot schools that had high accuracy rates in terms of alignment to the approved allocation. Of most importance, however, is simply that TnT has demonstrated its ability to generate real-time, comprehensive data on the accuracy of textbook supply. v. Overall Feedback About Distribution System Performance As a whole, all qualitative data collected by the project team confirms general support for TnT going forward. Respondents across the board have commented on TnT’s ability to help improve some elements of the distribution system performance, but also on the importance of TnT for making textbook supply performance more visible. The table below details comments during FGDs with different stakeholders about the usefulness of TnT. School Directors/Librarians’ Perspective SSCs’ Perspective · Save time, faster · Save time, faster · Save money · Enables them to know the amount of books and types received at their schools · Don’t need to make calculation for book request · Enables them to know if there are sufficient · Don’t need to bring the book request form to DOE textbooks for students · Book request reached MoEYS immediately 21 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 · More accountability · Receive information fast about the status of textbooks DOEs’ Perspective POEs’ Perspective · Save time, faster · Provides clear information · Save money · Makes book request fast · SDs don’t need to bring request forms to DOE · Transparent · Only need to enter a little information for book · Reduces pressure on POE from complaints request and reduce wrong calculation · Allows SSCs to know the book situation · More accountability · Make the book request and book distribution easier Suggestions collected by FGD participants pointed to a continued need for TnT and attention to the book supply chain. FGD members stated that they wanted to request that MoEYS deliver textbooks to schools on time in the right amount, right types of books, and with good quality books. The five people interviewed at central MoEYS level also expressed strong overall support for TnT and the positive impact it can have on the textbook supply chain. The Director of the PED commented on the usefulness of TnT for seeing, in real time, the progress of book requests and deliveries. He stated that once TnT is scaled up, his department would make more active use of the dashboard to track supply. The PDH official was particularly supportive of TnT for its ability to quicken data flow through shifting the paper-based system to digital. This is made easier by the increasing number of school staff now using the Telegram messaging service that TnT utilizes. C. Community Engagement One of the objectives of the project was to increase the engagement of School Support Committees (SSCs) in textbook supply at schools. Design features of TnT provided SSC members with the opportunity to use TnT to confirm book delivery and to conduct spot checks in classrooms. Results regarding this aspect of the pilot project are presented below. Level of Engagement: Results from all the various data sources confirm that the School Support Committees (SSCs) at pilot schools were involved and engaged in the textbook supply chain. Data from TnT can be analyzed to track SSC interaction with the system to understand the level of engagement by the school community. TnT generated data revealed that 60% of schools met the criteria for the incentives. ● % schools with at least one SSC member registered on TnT ● % SSCs that have counted the books received by their school ● % SSC that have conducted at least one spot check of book availability Overall, the data shows that 93% of pilot schools had at least one SSC member who registered on TnT at the beginning. In 89% of schools, an SSC member conducted a spot check of book availability at their school, to check on book access in the classroom (not to check on the delivery of the new 22 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 books). The data also showed that for 44% of deliveries during the first round of distribution (in January), an SSC member confirmed the delivery. It should be noted that each school on average received two deliveries during that round so this figure is 44% of deliveries. This figure went up to 70% for the second round in May/June. During project monitoring, the team met with 130 school directors and 76 SSC members to record their experiences in using TnT for distribution. All SSC members interviewed revealed that they had been contacted by their school directors informing them that the books had arrived. They were then asked to go to the school to count the amount of textbooks received and confirm the book receipt through TnT. Given that the baseline survey FGDs in 2017 revealed an absence of communication between school directors and SSCs, particularly regarding book distribution, this is a most positive finding. Another encouraging finding was that school directors waited for SSC members to arrive at the school and then unpacked the textbook boxes together. It was found that roughly 95% of surveyed SSCs declared that their school directors waited for them to arrive before unpacking the books. Technical Challenges with Using TnT: Findings from SSC FGDs demonstrate that they found interaction with the TnT platform relatively difficult. All SSC FGDs commented on the complexity of figuring out which were the correct links to use for the different disbursements. There were also comments regarding problems with limited ICT knowledge; forgetting how to use the system due to the long gap after training; poor internet connection; the need to reactivate their Telegram accounts and not receiving notifications because the Telegram account was no longer active; and, problems caused by changing their phone. Responses during FGDs with school directors corroborated these findings. Half of the FGDs confirmed that the SSCs needed assistance, usually from teachers, in filling out the book receipt form through TnT. FGDs with DOEs similarly suggested that over half of SSCs relied on people to help them. The table below summarizes comments during FGDs with school directors/librarians and DOEs regarding their view of the challenges of SCC involvement. School Directors’ Views on the Challe nges of SSC DOEs’ Views on the Challenges of SSC Involvement as Involvement as Expressed During the FGDs Expressed During the FGDs · SSC members do not have smartphones · SSC members do not have smartphones · SSC members have no/little ICT knowledge · Limited ICT knowledge · SSC members do not have time · It is difficult for SSC members to use the TnT system · SSC members do not have time · School directors do not want SSCs to know everything that is going on at their schools · SSC members do not value this task Improved Relationships Between SSCs and School Directors: SSC participants in the FGDs of both provinces were asked if the relationship between SSCs and school directors had changed or not as a result of TnT. All SSC FGD groups noticed that their relationship had become closer. SSCs stated that school directors invited them to count the received books and conduct book spot checks for the first time ever; they had never been involved in monitoring textbooks before. The four illustrative case studies paint a similar picture. All school 23 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 directors commented that a noticeable change was that SSC members were more involved in book distribution this year and they came to the school to check the receipt of books and to do a book spot check. D. Results-Based Financing (RBF) Efficacy As mentioned in the introduction section above, the project aimed to investigate RBF at two levels: centrally, with distributors, and at the community level with SSCs. Experiences of the project to date are detailed below. RBF for Distributors: During this current phase of the project, senior level managers from central MoEYS have been given access to the TnT dashboard to give them real-time information on book requests and delivery at all pilot schools. This includes key decision makers from within the Procurement Review Committee, the body responsible for determining payment to printers/distributors. The project found out that while 100% of the contract value is paid by the government to the contracted distributor/s before delivery as government funds must be spent by the end of the calendar year, the distributor/s has to pay 10% of the value of the contract into a bank account as a deposit. In the current government system, the Procurement Review Committee uses three criteria for determining when the 10% deposit can be released back to the printing house/distributors. The criteria are i. all receipts have been reviewed and found to be in order; ii. spot checks are made through phone calls to some schools to confirm receipt of books; and iii. school visits to a sample of schools, that are not selected on a random basis. This system is clearly riven with inefficiencies. In May 2018, the project team met with the Procurement Review Committee regarding the use of TnT for the 10% deposit release. This was a very high level meeting attended by the Secretary of State for Finance and Administration under MoEYS as well as the Director General of Finance and Administration, the Director General of General Education and the Directors of the Primary Department and the Curriculum Development Department, among others. Members of the PRC raised a number of concerns about TnT, for instance that we were requiring the contractors to perform tasks not listed in their contract. The Procurement Review Committee requested evidence in 2019 that the TnT design functioned effectively in practice, that school directors and others were able to use it, and that it generated reliable and comprehensive data. Overall, however, there was a successful outcome to the meeting. The Procurement Review Committee agreed, in principle, that once TnT is used at scale, and once it is explicitly stated in distributors’ contracts, TnT data could potentially be used as an additional criterion for releasing the 10% deposit to distributors. Furthermore, the five MoEYS officials who were later interviewed in September 2019 as part of the endline process were asked their opinions whether, once TnT is used nationally, TnT data will indeed be used as an extra criterion for releasing payment of the 10% deposit to distributors. The majority responded positively and said the Procurement Review Committee should have full access to the TnT dashboard/data. However, the majority also stressed the importance of ensuring TnT data was reliable before asking the PRC to use it as an extra criterion. Overall indications were that although they do not recommend using TnT at this pilot stage, in the future they support its use for determining distributor payment. Of particular encouragement was the support from the official interviewed from the Procurement Unit, who had been less positive of the benefits of TnT during earlier stages of the project. During his endline interview, he expressed his support for the use of TnT for managing and paying the contracted distributors. He stated how TnT would allow him to evaluate the performance of the contractors’ service and it would help the Procurement Unit to manage the contracts better. 24 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 It has been interesting to note throughout the life of the pilot that TnT has stimulated awareness and tense discussions within and across MoEYS’ departments about the value of using TnT for deposit release. To quote the Director of the Primary Education Department, HE Chan Sophea, when asked if MoEYS should continue with TnT after project end: “There can be no going back now.” In general, the project has received a high level of support by senior MoEYS officials – the Minister, Director Generals of Primary and General Education, Director of the Primary Ed Department, among others. Their level of engagement has been remarkable. They have cited intense frustration with the current textbook situation and are most welcoming of the potential TnT has to help. Lastly, it is important to point out that the Publishing and Distribution House (who is mandated to receive 50% of the book printing/distribution contract each year) welcomes the visibility and the results-based financing element to such an extent that they have committed to pay for TnT running costs themselves from 2020 to make sure TnT continues to operate. The PDH have said repeatedly that they welcome TnT for its ability to demonstrate their performance in book delivery and to eliminate inaccurate criticisms of their service. This was again mentioned by the Deputy Director General during endline interviews. Overall, it can be concluded that in this pilot the TnT platform has demonstrated to MoEYS its ability to generate the data needed for performance-based contracting of printers/distributors. There is now a growing recognition within MoEYS that TnT could be used for this purpose once TnT is applied at national scale. RBF at Community Level: As mentioned earlier, the project tested out incentivizing SSCs to be involved in textbook supply by asking DOEs to award certificates to SSCs who fulfilled all of the following criteria: registered on TnT, confirmed textbook receipt and conducted a spot check. In total, 60% of SSC’s fulfilled these three criteria and are eligible to receive a certificate. All ten DOEs have confirmed that they will print out certificates for these SSCs and these will be presented during school opening ceremonies. During FGDs, participants were asked whether incentives should continue to be provided to SSCs for their involvement in the future, and specifically whether certificates were an appropriate incentive. General agreement was found among POEs, DOEs, and school directors that motivating SSCs to be involved in textbook supply by offering certificates was a good approach. SSCs were also asked during FGDs whether the opportunity of receiving certificates influenced SSC behaviors. All SSC FGDs responded that certificates would encourage them to engage more and be actively Example of a certificate awarded to SCCs who involved. fulfilled all criteria Questions were put to school directors, DOEs and the POE whether it is useful to have the SSC involved in checking book supply. All responded positively, citing reasons that included accountability, transparency, and accuracy. DOEs and POE staff in target provinces were asked about the willingness of DOEs and the POE to continue to use their own budgets to issue certificates for SSC. All responded affirmatively. Endline interviews with central MoEYS leaders also indicated support for involvement and incentivization of SSCs. There was overall agreement among all five interviewees that once TnT is used at national scale, SSCs should still be a user of TnT and should help check the book supply at 25 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 schools. They supported the current piloting of certificates as incentives, but also mentioned alternatives such as encouraging school directors to add an incentive if possible. The Director of the PED referred to the upcoming change of title from SSC to School Management Committee (SMC) and the possibility of more clearly defining the SMC’s role in textbook supply chain/confirming book receipt into their ToR. However, none of the MoEYS leaders at central level could offer suggestions on how MoEYS would train and support SSCs to use TnT once at national scale, and in particular where the training budget would be sourced from. 26 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 5. Next Steps for Track and Trace in Cambodia Based on the success of the TnT pilot and the extremely close government involvement, the Minister of MoEYS stated on April 25, 2019 that he wants TnT to be used at scale in Cambodia, and that MoEYS will pay associated license/running costs (approximately $13,000 per year at national scale at current estimates). It was later agreed that funds will come from the Publishing and Distribution House (which is a state-owned enterprise under MoEYS), at the offer of the Director of the Publishing and Distribution House, and not from regular MoEYS funding streams. This will make payment for the running costs of TnT much more straightforward. The Minister has asked WEI to find funds to pay for the necessary technical refinements to the TnT platform; to develop a national training package and training delivery model; and, to pay for a limited amount of the training that will be required for roll-out. Without this additional funding, the ministry will not be able to take over and scale up Track and Trace. Modifications and streamlining of the digital TnT platform are essential for MoEYS to use it at scale. This is based on a number of lessons learned during the pilot project. 1. Firstly, during the pilot MoEYS instituted some changes to its textbook requesting and distribution process, partially as a result of the project shining a light on the current inefficiencies. However, the original TnT system was not able to support all of these changes. During this next phase, we will refine the system to make it more flexible to enable MoEYS to make changes in the process without requiring a future system redesign. 2. Secondly, the TnT platform designed for the initial phase of the project aimed to provide visibility into each step of the supply chain. As we move towards scaling with MoEYS’ own budget, the system needs to be simplified to reduce system and training costs. Therefore, only the critical steps of the supply chain will be maintained in the new system. 3. Thirdly as the Publishing and Distribution House (PDH) is the specific entity taking on management/ownership of TnT, the requirements of the PDH need to be considered, such as printing of receipts, while still also providing appropriate visibility for other MoEYS leaders/departments. The needs of the PDH need to be included in the modifications to the platform. If additional funds can be secured, the plans are to use the period October 1 st 2019 to December 31st 2020 to work side-by-side with MoEYS and the PDH to finalize the revised design of the TnT digital platform, determine the scope and locations for scaling, and design a feasible and affordable training package that will enable all users to engage with TnT appropriately across the nation. A developer company will be contracted to make refinements to the TnT platform, and it will be ready to go live from July 2020. From that point onwards, the PDH will fund the ongoing running costs of the platform. The project will work with MoEYS to ensure all schools across the country enter their book requests into the TnT platform in July-August 2020 which will inform the quantities allocated to schools for the 2020-2021 school year. Towards the end of 2020, TnT will then be used to track delivery to schools in a defined number of provinces (likely 5). School directors will choose provinces according to data collected by the PDH regarding smartphone usage. After the project has ended, MoEYS will lead further incremental rollout by itself in 2021 and will manage the platform. WEI and JSI held extensive consultations with MoEYS leadership in May and June 2019 regarding the next phase. The key decision points from these consultations are listed in the box below: 27 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Based on the consultations held with the PDH and MoEYS National Consultative Group in May and June, the current understanding of the revised TnT platform design is illustrated in the diagram below. 28 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Use of Results-Based Financing (RBF) Going forward, TnT will provide the data to enable the Ministry of Education, Youth and Sport (MoEYS) to measure the performance of the textbook supply chain and track improvements and achievement of targets. The project will encourage MoEYS to use TnT data to hold printers/distributors accountable for delivering the right books to the right schools at the right time and in the right condition. As noted already above, the Procurement Review Committee, Procurement Unit and other senior MoEYS leaders have indicated that once TnT is used at scale, and once it is explicitly stated in distributors’ contracts, TnT data could potentially be used as an additional criterion for releasing the 10% deposit to distributors. This issue will remain at the forefront of discussions with MoEYS throughout this next phase of the project. All relevant MoEYS personnel at central level will be provided access to the TnT dashboard to enable them to utilize the data they need for determining when the 10% deposit can be released to distributors. Overall, WEI will continue to work towards achieving the first performance-based contracting of distributors using TnT. TnT has already demonstrated its ability to generate the data needed for such performance-based contracting and there is a growing momentum within MoEYS for this to be used once TnT is applied at broader scale. Secondly, the project will explore with MoEYS whether it intends to continue to engage the school community (primarily School Support Committees) with the revised TnT platform once MoEYS take it over next year. As presented earlier in this report, endline FGDs with school directors, SSCs and DOEs all showed support for the SSC role in TnT. Consultations with MoEYS, as part of the endline interviews for the pilot project, similarly suggested that central MoEYS is supportive of maintaining the SSC role with TnT. However, of concern is that the leaders from MoEYS who were interviewed could not provide feasible suggestions for how the training of SSCs might be conducted and funded once TnT is used at national scale. Further consultations with MoEYS under the new project (in October 2019) will confirm whether MoEYS has the necessary resources for training SSC and intends to maintain SSCs as users of TnT. If so, the new platform and training package will be designed accordingly to enable this, and the dashboard will be configured to generate reports that indicate which SSCs have fulfilled the three required criteria in order to obtain the certificate incentive. The current phase of the project has most certainly demonstrated that SSCs are keen and active when given a chance to be involved in textbook supply and that local DOEs are willing to use their own budget to reward/incentivize SSCs for this involvement. 29 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 6. Conclusions and Policy Implications for Track and Trace in Cambodia Overall, it has been seen that the Track and Trace pilot has achieved what it set out to do. The project has demonstrated the potential for TnT to make the school textbook supply process more efficient in three ways: 1. Book requests - Schools can now report timely data and TnT assists in accurately projecting book needs for budgeting and procurement by the central ministry 2. Tracking “real-time” distribution of textbooks and providing visibility of issues for the government at all levels 3. Engaging school communities to confirm books arrive and conducting classroom spot checks post-delivery to ensure textbooks are available Through using TnT, the central Department for Curriculum Development (DCD) at MoEYS had received book request data from all of the pilot schools within two weeks, rather than 6 months that is typical with the paper-based system. All users of the system, and MoEYS at senior level, have expressed much satisfaction with the requesting functions of TnT. TnT has demonstrated its ability to generate useful data, that allows users at MoEYS to see in real time the timing, accuracy, condition and location of book deliveries. For instance, it has produced indicators such as % shipments received at DOEs; % school directors that received all shipments; % of deliveries to schools by month; % school directors’ receipts that match approved quantities in the allocation list; and % of total books received by school directors compared to approved quantities in the allocation list by grade and subject. Such data allows MoEYS to identify some of the bottlenecks and issues that are preventing students from receiving their textbooks. For instance, data from TnT showed that while overall the printing houses delivered the correct total number of books to pilot schools (99% of the approved amount in total), these books were not always allocated according to the approved amounts for the individual schools (only 94% of school director digital receipts on TnT matched their approved amounts). This could have been an issue at the district level where books are allocated to schools. Overall, TnT data such as this will contribute to MoEYS’ ultimate goal of ensuring every student has the right textbooks at the right time and in the right condition. Feedback from users has been positive. The majority of school directors were able to use TnT unassisted, although some difficulties have been reported. There was a clear need for training on the various features to be provided just before users engage with that particular aspect of the system, and it also became apparent that some elements of the system should be simplified. Both of these aspects will be addressed in the next phase of the project. The project also proved able to engage School Support Committees. The majority of schools had SSC’s who utilized TnT to record book receipts and conduct classroom spot checks. However, a significant proportion found the system quite hard to use and this might be problematic when TnT is taken to scale and MoEYS cannot afford to provide in-person training to SSCs. A reliance on digital training tools and guidance is less likely to be feasible for SSCs than for school directors. The graphic to the right summarizes the overall benefits of TnT. 30 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 These benefits have been recognized by MoEYS, as made evident by the Minister’s declaration to scale up TnT nationally, and the offer by the PDH to pay for the running costs. In addition, it appears likely that the Procurement Review Committee will consider using TnT data as one of the criteria for determining deposit release to distributors once TnT is used at scale. A number of more detailed learnings and observations are provided below about the experience and outcomes of the project to date. Strong Government Engagement Was a Crucial Factor The creation of the National Consultative Group at the start set the tone for the remainder of the pilot. The project gave this group a genuine (not token) role - participating in the administration of the baseline; providing extremely detailed input into the design of the entire platform; assisting in the development of training materials and facilitation of the trainings; and, field monitoring of requests and distribution. Meetings with the whole NCG were organized at strategic points, and it was members of the NCG who presented the platform to the senior leaders of MoEYS and advocated for its nation-wide adoption, rather than the project staff. The TnT Platform Needs to be Simplified for Nation-Wide Use and Government Takeover It has become evident to the project team that the MoEYS textbook supply chain process keeps changing. Even within the project time frame, multiple changes have been made by MoEYS to final point of delivery, the time frame of requests, content of the requests, and so forth. Such changes can have serious consequences for the functioning of the TnT system. There is a need to develop a much simpler platform that does not require constant re-development whenever MoEYS tweaks the process (for instance, MoEYS suddenly decided that cluster school directors not District Offices of Education should distribute books to school directors, but TnT had not been structured to allow for data generation at the cluster level). There are trade-offs between visibility at every stage of the chain, and long term durability of a TnT platform that can accommodate any further changes to the textbook supply process that MoEYS may decide on. Simple tracking of dispatch from central warehouse and receipt at the school reduces costs and may be enough to achieve the majority of the objectives. The revised design of TnT for 2020 will be streamlined. For instance, TnT will no longer track each step in the delivery process. Instead, the system will track what books leave the warehouse and then what is received at 31 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 the school level, but will not track what is received at intermediary levels such as district or cluster schools. The PDH will notify the system when the deliveries are dispatched using the dashboard and school directors will confirm quantities received via the chatbot after each delivery. The receipt form on the chatbot will be simplified by removing the automated receipt number. If the SD received two shipments from different suppliers around the same time, they can use the same receipt form to confirm quantities received. This reduces the need for the smartphone application, instead of bar codes the system uses only chatbots (Telegram) that connect to a web-based information system (Salesforce). This not only simplifies the technical changes that would need to be made to the TnT platform if/when MoEYS makes further adjustments to the textbook supply process, but it also addresses some of the difficulties found in using the system. For instance, as noted earlier, FGDs and surveys showed that school directors found the use of TnT during disbursement to be overly complicated with too many links for the different book deliveries, as each delivery had a different link to a receipt and as different companies were delivering at the same time, this became confusing. The TnT Pilot Had a Limited Impact on Book Access and Supply Chain Performance in the Short Term This should come as little surprise, and the project has been clear from the outset that there are critical weak points in the supply chain that TnT is not able to address, especially the bottlenecks around procurement. As noted earlier, due to a particularly fractious year between the two distributor companies and MoEYS, contracts were only signed at the end of 2018/early 2019. Eighty percent of books were delivered almost three months late, and 10% of books were still being printed and delivered in June - which is only two months before the end of the school year. Naturally, this meant that book shortages persisted, and schools ended up distributing old and damaged books to students. However, we have seen that TnT improved requests significantly. Over the longer term, the improved timeliness and accuracy of request data due to TnT will have a knock on effect of enabling book allocation to be based on recent request data/school needs. This, in turn, should reduce book shortages. In addition, the demonstrated ability of TnT to make visible, in real time, the very poor status of book deliveries provides a significant impetus to improve. Results-Based Financing for Distributors Will Take Time Consultations with senior leaders of MoEYS, such as the Minister, various Director Generals and Directors of departments, from before the project right the way to its close, all revealed a high desire to make textbook distributors more accountable and to have increased visibility and accountability into the textbook supply chain. This gave the project team much optimism that TnT data would be utilized in determining contractor payment, and it continues to do so. The Minister’s declaration that MoEYS will take over TnT from 2020 is a clear reflection of their recognition of TnT’s value. Nevertheless, project experience has made it clear that formal use of TnT for RBF of distributors is going to take time. Procurement and financing processes of the Ministry of Economy and Finance, and the Department of Administration and Finance within MoEYS hold sway and are not easily changed. They also do not facilitate easy adoption of RBF, particularly given that 100% of the contract value is paid by the government to the contracted distributor/s before delivery as government funds must be spent by the end of the calendar year. Admittedly, the distributor/s has to pay 10% of the value of the contract into a bank account as a deposit, and this does present a window in which RBF based on disbursement can operate. Generally speaking, it can be said that the reaction of the Department of Administration and Finance to TnT has been fairly muted. However, grounds for cautious optimism were given by the verbal agreement in May 2018 by the Procurement Review Committee that once TnT is used at scale, and 32 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 once it is explicitly stated in distributors’ contracts, TnT data could potentially be used as an additional criterion for releasing the 10% deposit to distributors. A final interview with the Chief of the Procurement Unit was also most encouragingly positive, with his reiteration that TnT would be useful for managing and paying distributors. The overall impression is that things are moving in the right direction. While it may take some time, and it will need robust evidence from TnT that its data is reliable once used at scale, the general momentum within MoEYS is one of change. The comment from the Director of the Primary Education Department, HE Chan Sophea, “There can be no going back now,” when asked if MoEYS should continue with TnT after project end, is a good summing up of the opinions found by the project team among MoEYS. TnT Could Help Spur a Shift to Digital Receipts TnT offers MoEYS and the Ministry of Economy and Finance the opportunity to shift to digital book receipts at every level of the distribution chain. The current (costly, inefficient, easy-to-falsify) 4-ply duplicate paper receipts are made obsolete by having a digital system (on which digital signatures, geo-location and time stamps are all enabled). While there were many within MoEYS (the Department of IT, the Textbook Supply Office, etc.) who were very keen, the Procurement Unit and the PDH appeared flatly opposed to this shift to digital, at least during the pilot phase of TnT. This is perhaps natural in that TnT was only piloted on a small scale and it had not yet proven its ability to generate reliable data and evidence of book receipts on a larger scale. During the project, the Procurement Unit did not even allow the use of TnT-generated paper receipts rather than their own paper receipts among the pilot schools. However, there are some reports that the Ministry of Economy and Finance is in the process of beginning to digitize receipts more generally in other sectors, using on-screen signatures, which would be a most promising development. If this is the case, TnT will provide a valuable model to the government, not just in terms of technical design, but also in terms of training and roll out at school level. Training Considerations Will Present Challenges for Wide-Scale Use of TnT Findings from the FGDs, surveys and case studies have made it clear that users of TnT need ‘in-time’ training and support immediately before they are expected to use the system, and ongoing access to guidance throughout. This presents a challenge because school directors, for instance, are required to use TnT during requests and then there is a long gap until they need to use it again during the delivery process. However, it is not cost efficient to provide face-to-face training more than once, if ever, when rolling out TnT nationwide. There are obvious ways to address this, including the use of technology as well as ‘piggy backing’ short trainings/reminders about how to use TnT during other trainings that MoEYS might be providing to school directors. WEI is one of the pioneers of the use of technology in education and training, both in Cambodia and globally, and will put this expertise to good use during the next phase of TnT. We will work with MoEYS to design and trial the use of e- learning modules, videos, chatbots, social media and instant messaging services for ‘spaced’ learning delivery in small chunks, amongst possible other strategies. Challenges Associated with the Involvement of School Support Committees The endline surveys, FGDs and consultations found general support for having SSCs as users of TnT. What is more, TnT data shows that 93% of pilot schools had at least one SSC member who registered on TnT; 50% of deliveries were confirmed by the SSCs for the first round of distribution in January and 75% for the second round in May/June; and, 89% of SSCs conducted at least one spot check of book availability at their school. However, this is tempered by the fact that this level of engagement was only created by direct contact through Telegram and other means with SSCs to remind them to use TnT and to offer guidance. Such direct contact is unlikely to be replicable by MoEYS when TnT is rolled out at greater scale. Further, endline findings are clear that SSCs found the use of TnT to be 33 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 difficult, and that their level of digital skills are low. The greatest barrier, however, is likely to be training. While technology might offer a possible route for training school directors, it is less likely to be appropriate for SSCs who, our research shows, have less digital access and competence. So far, MoEYS have not offered the project team feasible and affordable suggestions for how SSCs might be trained nationally to use TnT. For the next phase of the project, it has been agreed that this will be investigated further and while SSCs will be given the opportunity to use TnT, it will not be required. 34 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 7. Analysis and Recommendation for Design and Pricing of TnT in Other Contexts A number of lessons have been learned during the design and piloting of Track and Trace in Cambodia which can help similar initiatives elsewhere. These, as well as the results of the project overall, have been shared widely within Cambodia and globally through photo story features, a video, podcast, posts on social media and fact sheets all distributed through strategic sites on the Internet. A number of high profile conference and webinar presentations have also been made, including a presentation with the Global Book Alliance and an ICT roundtable at the Comparative International Education Society conference in San Francisco in April 2019; participation in a roundtable with the Global Reading Network in Washington DC in February 2018 and a demonstration at the All Children Reading: A Grand Challenge for Development EdTech Open in October 2019. WEI and JSI have also become active members of USAID’s Global Reading Network’s Special Interest Group on textbook supply chains. A selection of our lessons learned are provided below. Use of Smartphones and Chatbots In Cambodia, smartphones and Internet access have been spreading widely in recent years, and the project’s baseline research indicated that 75% of school directors had access to data through their smartphones and were already using it for accessing social media, instant messaging apps and other purposes. For that reason, TnT was built around the use of Internet/data-enabled technologies - a chatbot, online forms, etc. The pilot also operated on the assumption that users did not need to be reimbursed for their use of TnT because they were purchasing their own data anyway, and TnT would A school director uses his smartphone to access the TnT platform not utilize much of their data. It turned out that such assumptions were correct. At least 85% of the school directors could use the technology unassisted in TnT pilot areas. Workarounds were also found for the SSCs, who our baseline showed rarely used smartphones (20%) and never used SMS. By encouraging SSCs to find an assistant who did have digital access and competence (such as a teacher at the school, or an older student), the project found that 93% of SSCs were able to register as users on TnT, albeit with support. In general, then, it can be said that the use of Internet/data-based technologies is feasible in contexts such as Cambodia. This, in turn, has the advantage of overcoming costs associated with the use of SMS or IVR in other TnT models elsewhere. Annual License Costs/Running Costs Can Be Very Modest The design of TnT used during the pilot is estimated to cost USD$15,000 if used at national scale for annual or monthly licensing costs for the various component parts of TnT (Salesforce, Taroworks, FlowXO, FormAssembly). The revised and simplified design of TnT is likely to cost between USD$12,500 if for-profit pricing is applied to just under USD$5,000 if it proves possible to get non- profit rates from Salesforce for a government owned company. Such modest costs are mainly due to the use of Internet-based solutions such chatbots through Telegram and on-line forms. It is because we have been able to keep running costs low that MoEYS, and in fact the Publishing and Distribution House, has stepped forward and declared that it is prepared to take over the running costs of TnT and roll it out at scale from 2020. 35 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Need for Lobbying to Salesforce and Other Providers for Reasonable License Rates for LMIC Governments At present, WEI and JSI are in discussions with Salesforce about the licensing costs for TnT once it is managed by MoEYS and rolled out next year. Salesforce’s current pricing model offers discounted/free licenses to non-profits but charges governments full corporate rates, which are significantly higher. In the case of Cambodia, it would be most shortsighted of Salesforce to charge MoEYS corporate rates. Using commercial based software as a service has the advantage of a robust software that is well maintained and supported. Salesforce has a strong community and many options in terms of added functionality. The adoption of TnT by MoEYS offers Salesforce a strategic opportunity for recognition, and is one that might be replicated elsewhere. Lobbying to large technology companies such as Salesforce would have more effect if done with the help of entities such as the World Bank, USAID and others who are promoting textbook access globally. TnT Proved Particularly Useful for Book Requests During the initial baseline and feasibility assessments of the project it became clear that one of the major bottlenecks in textbook supply was the existing book request process. It was therefore decided to add in book request functions to TnT. This is not common among Track and Trace systems used in other LMIC. The requesting functions that were built into our TnT were designed to improve the timeliness and accuracy of book projections and improve the efficiency of preparing budgets and procurement plans. It had been found during our baseline that the process of book requests from schools to the central ministry (the Department for Curriculum Development, DCD) usually takes about six months; that school directors are expected to calculate complicated projections two years in advance; paper forms travel from schools to districts to province to central ministry; and, that the reports are then manually entered which is time consuming and prone to errors. TnT addressed these challenges in a number of ways: 1. School directors or librarians are not required to calculate projections two years in advance using complicated formulae that are open to error. TnT calculates the projections based on current and past enrolment data and usable books in stock data. 2. School requests are not entered manually by DCD staff, thus reducing time and errors. DCD staff are able to view national book requests immediately after schools submit data. 3. DCD staff can use those book request numbers to improve budget accuracy and make objective adjustments to book allocation lists During the pilot, all request data was submitted within two weeks, which offered the potential for the DCD to use the figures for preparing the budget for the following year. In the existing paper-based system the data that school directors report can only be used for procurement of textbooks two years in advance because of this long delay in collecting and compiling all the reports at national level. We found that it was these book request features of TnT that proved most popular among all users. Indeed, the PDH have stated that it is this functionality that spurred Mr. Samchivien, a school director in Banteay Meanchey province, uses his smartphone to the PDH to offer to take over the payment and request textbooks through the TnT system management of TnT in general because the Minister has recently appointed the PDH to be in charge of textbook requests nationally. The PDH wanted a more efficient, and particularly more timely and accurate system for collecting book requests. 36 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Cost-Benefit Considerations A basic cost-benefit calculation highlights the value of TnT in Cambodia. Based on the conservative assumption that currently at least 2% of the contracted textbook allocation quantity does not reach its intended destination, there would be a cost saving to MoEYS of $206,000 out of an annual budget of just over $5 million on textbooks. This makes the anticipated outlay of between USD $13,000 and USD $5,000 on TnT running costs seem like a wise investment of money. Trade-offs Between Full Visibility and Feasibility of Sustainable Roll Out and National Ownership The final TnT that was designed for the pilot was a complicated system with many steps, features and users. This was deliberate, to allow full visibility into each part of the supply chain and thus to allow MoEYS to pinpoint precise weaknesses in the supply chain. However, the disadvantage is that every time MoEYS slightly adjusts book supply chain processes (date of requests, data required, delivery points and personnel, etc.), the TnT platform requires back end changes that can only be done by skilled and costly developers. Furthermore, there are associated increased training costs and overall running costs related to having a more complicated system, and as our end line showed there were quite a proportion of users who found the system difficult to use. For the next phase in Cambodia - national scale up and hand over to MoEYS - the system is undergoing a redesign to make it more streamlined, but at the cost of reducing the level of visibility into each small step of the delivery process. Applications of Results-Based Financing (RBF) To date, use of RBF in education supply chains in LMIC has been limited and this project has generated some experience around the role of RBF in increasing school community engagement and in improving the distribution of books. At the community level, the project has shown that, at least in Cambodia, School Support Committees are keen to be given a role in textbook supply, and embrace that role when the opportunity is given. Although the project offered certificates to School Support Committees that are fully engaged with TnT (registering, going to school to count book received, conducting a classroom spot check a few weeks/months later), it appears that the chance for meaningful engagement with the school was incentive enough. Overall, 60% of the 416 SSCs in the pilot filled all three criteria for receiving a certificate. How the results might turn out when TnT is used at broader scale remain to be seen, however. For the use of RBF at central level, the project has demonstrated that a TnT system can generate useful data for holding distributors accountable. While it is clear that bringing actual change to well- established government contracting processes takes time, our pilot experience in Cambodia is that the demonstrated capabilities of TnT have resulted in discussions at the top levels of the Ministry of Education and positive indications that changes to payment criteria for distributors will occur once TnT is used nationwide. In addition, the project has already seen a number of important changes to the textbook supply chain processes during the life of the project. It is highly likely that the increased supply chain visibility from TnT and its highlighting of where the challenges are along the supply chain have already contributed to these policy reforms to improve the education supply chain in Cambodia. 37 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Annex 1: Track and Trace Evaluation Plan What Who When Tool Details Access Measurements Textbook Policy Checklist TnT Team End March - Adapted digital NEP In the same schools as baseline - 42 early May 2019 Textbook Policy Checklist Results will be analysed in comparison to baseline data Spot checks TnT Team After books Simple project-made spot In a small random sample of approx. 15 schools out of the distributed check tool 416 target schools (approx. March - Will investigate if the newly delivered books are being July 2019) used in classrooms. Is not about ‘access to books’ generally. Just about access to the new books SSC classroom visits SSC+ After books TnT platform TnT platform will notify SSC+ to do the visits and provide distributed the form. Data will be generated through TnT system and can be viewed on dashboard/reports Distribution Measurements Data on requests, N/A ongoing TnT platform Generated through TnT disbursements, timing, Analysed by TnT team accuracy, completeness Focus groups with SDs, SSCs, TnT Team After books Similar FGD tools as used Same 42 schools/districts as baseline teachers, and DOEs distributed in baseline - but adapted Results will be analysed in comparison to baseline data (approx. August to investigate changes as a 2019) result of TnT Interviews with POE and TnT Team August- Interview tools to be made Results will be analysed in comparison to baseline data central ministry September 2019 by project Case studies TnT Team May 2018 Case study template made 4 SDs. 2-3 of whom identified as having problems, and October 2018 - by project remainder identified as not having problem. March 2019 Same people interviewed at least twice. Once during requesting period, and again during distribution. Written interview as well as video footage and photos taken Community Engagement Measures 38 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 What Who When Tool Details Data on SSC involvement: % N/A ongoing TnT platform Generated through TnT SSC with 1 person Analysed by TnT team registered, % SSC count books, % SSC spot check classrooms Focus groups with SDs, SSCs, TnT Team After books Questions incorporated Same 42 schools/districts as baseline teachers, and DOEs distributed into the FGDs to Results will be analysed in comparison to baseline data (August 2019) investigate SSC involvement RBF Efficacy Focus groups with SDs, SSCs, TnT Team After books Questions incorporated Investigates RBF component effect on SSC behaviour and DOEs distributed into the FGDs to (August 2019) investigate RBF efficacy Interviews with central TnT Team Aug-Sept 2019 Interview tools to be made Investigates whether TnT was used to determine release ministry by project of 10% deposit to PDH Project Implementation Monitoring Training attendance lists TnT Team Each training Participant list Trainers Training evaluations TnT Team Some trainings Questionnaire Trainers Book request school visits / TnT Team May 2018 during Book Request School Visit Digital tool. Enabled with on-screen signatures, GPS and spot checks Trainers book request Form time stamp features. ‘window’ Struggling/potentially problematic schools chosen for visits based on TnT data and trainers’ recommendation Other schools chosen on a random spot check basis Book distribution school TnT Team January - July Book Distribution School Digital tool. Enabled with on-screen signatures, GPS and visits /spot checks Trainers 2019 during book Visit Form time stamp features. distribution Struggling/potentially problematic schools chosen for ‘window’ visits based on TnT data and trainers’ recommendation Other schools chosen on a random spot check basis 39 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Annex 2: Illustrative Case Studies Track and Trace (TnT) is a two-year program implemented by World Education, Inc. and supported by the World Bank and Ministry of Education, Youth and Sport (MoEYS). The program aims to improve the supply chain management of textbooks in Cambodia and is being implemented across Banteay Meanchey and Prey Veng provinces. TRACK AND TRACE: ENABLING SCHOOL DIRECTORS TO LEARN, LEAD & SUPPORT IMPROVED EDUCATION AT PRIMARY SCHOOLS IN CAMBODIA Banteay Meanchey Province Mr. Samchivien is the School Director at Kleng Por Primary School in Svay Check district of Banteay Meachey province. He has worked and served the school since 1992, and has been the School Director for the past 6 years. “Being a school director, I aim to better my school in terms of the quality of education, as well as the learning environment and school infrastructure. I support activities that improve the quality of learning in the classroom, so that students will be able to pass from one grade to another,” said Mr. Samchivien. Mr. Samchivien was excited to use the Track and Trace (TnT) system to send textbook requests. This Mr. Samchivien confirming the receipt of his school's is the first time that he used a mobile device to textbooks through the TnT system submit textbook procurement requests for his school. At first, Mr. Samchivien struggled to use the This year, Mr. Samchivien was able to use the TnT TnT system, because the process of submitting system not only to request textbooks for his school, requests was very different from what he had done but also to con firm their receipt. “Using the TnT in the past. “At first it was quite hard to use TnT system to confirm textbook receipt is faster and system, but later when submitting book requests, I saves time,” Mr. Samchivien said. “Distributing books found it was both easier and faster.” through the TnT system is more accurate and accountable than the old system,” he added. Mr. Samchivien used to have to calculate the number of textbooks his school needed manually. Now, the He also explained that, this year, the School Support TnT system automatically does it for him. “Doing the Committee was more involved in the book book request this way is useful, because I can see the distribution process than in the past. calculated results in a Telegram message. All levels The Track and Trace project is working to ensure can also see the book requests at the same time,” he that students are equipped with the learning explained. materials they need to succeed in the classroom. It is also helping school directors, like Mr. Samchivien, upgrade their digital literacy skills and spend less time on administrative tasks. Thanks to TnT, Mr. Samchivien now has more time to improve the quality of education at his school. Mr. Samchivien requesting textbooks through the TnT system. 40 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Prey Veng Province Since 2017, World Education, Inc. and John Snow, Inc. Thanks to the training they received, Mr. Vesna and Mr. (JSI) have been working in close collaboration with Rith were successfully able to use TnT to submit their the Cambodian Ministry of Education, Youth, and school’s book request. Sport to develop and pilot the Track and Trace digital platform (TnT). TnT is a technology solution that Mr. Vesna was surprised at much the TnT system simplified enables ministry officials, partners, and distributors the book request process. “Submitting book requests to track in real-time the ordering and distribution of through the TnT platform is much easier than the old paper textbooks. This helps to ensure that textbooks reach method,” he said. He explained that the paper system was the schools they were destined for in a timely complicated and prone to error. When using the paper manner and in the right condition. system, “sometimes the District Office of Education (DOE) might unintentionally type the wrong number or make a mistake when calculating the number of books requested, which leads to errors in the number of books our schools receive,” Mr. Vesna explained. “The TnT system is much more accurate because it automatically calculates the number of books requested,” he said. During our visit to his school, Mr. Vesna was proud to show off his knowledge of the TnT system. Mr. Rith and Mr. Sinath (pictured below) skillfully demonstrated how to use TnT to submit a textbook request without any additional support or assistance. We are thrilled that the TnT platform can help Mr. Vesna For the past 6 years, Mr. Sen Vesna (pictured above) improve the school that he cares so deeply about. has served as the School Director of Oudom Samakom Primary School in Prey Veng province. Mr. Vesna has worked hard to improve his school’s facilities, often working closely with the local community to accomplish his goals. Mr. Vesna is very dedicated to his school and is respected by his teachers, students, and community. Mr. Vesna’s school was one of 416 schools selected to use the TnT platform to complete textbook requests in May 2018. Before the platform went “live” in May 2018, Mr. Deputy School Director Mr. Rith (left) cooperates with Mr. Sinath Vesna and his Deputy School Director, Mr. Rith, (right) to request textbooks using the TnT system. attended training about how to use the TnT platform. All School Directors from all 416 pilot schools attended this training on how to use the TnT platform to request textbooks. Thanks to the training they received, Mr. Vesna and Mr. Rith were successfully able to use TnT to submit their school’s book request. 41 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Banteay Meanchey Province Mr. Kheng Chanthol has served as the School Director of O Sampor II Primary School in Banteay Meanchey province for the past 11 years. Mr. Chanthol loves his work as a School Director. “The best thing about my job is that I get to share knowledge and help children become good citizens so that they can contribute to the success of our country,” he said. Mr. Chanthol is always looking for ways to improve himself and his school. “I would like to develop my professional skills further by receiving training on school leadership and management. Then I would be Mr. Chanthol using his smartphone to request textbooks able to lead and manage my school more eff ectively,” he said. He would also like to sharpen his basic Overall, 100% of the 416 pilot schools were able to computer and digital literacy skills. “I would love to be successfully submit book requests in May using TnT within able to do more of my administrative work on the the approved two-week timeframe. The TnT system then computer. I would be much faster, and then I would made the data available for immediate use by staff from have more time to spend on other things,” Mr. the Department of Curriculum Development (DCD) at the Chanthol added. central level. When he was informed that his school would be using “I like using the TnT system because it is faster and easier the Track and Trace system to submit book requests, than requesting the textbook by paper. It saves times and Mr. Chanthol was thrilled. “The TnT system saves so it is easy to correct the data if you make a mistake,” Mr. much time,” he explained. Chanthol declared. “The system even sends you a message through the Telegram app after you submit a request so Mr. Chanthol enjoys using technology and is very that you can check for errors,” he added. comfortable with his smartphone. He found navigating the TnT platform to be straightforward and The Track and Trace system improves the textbook supply simple. “I am more comfortable using the digital TnT chain by providing timely data and accurate projections of system to submit my book requests than I am using book needs for budgeting and procurement purposes. The the paper-based system,” he said. quick and easy two-step process for requesting textbooks means that school directors, like Mr. Chanthol, are “I was able to use the Telegram and the TnT system spending less time on administrative tasks and more time to submit my book requests on the first day of the focusing on the more important issues at their schools. request period. I am very proud of that,” Mr. Chanthol shared. 42 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Banteay Meanchey Province Mr. Mork Pov, 51-years-old, is the school director of Mr. Pov is always looking for opportunities to improve Koup Thom Primary School in Banteay Meanchey his school. He also enjoys finding ways for his school and province. Mr. Pov has served as School Director for the his community to work together. Just recently, for past five years. He enjoys his role and is passionate instance, Mr. Pov was able to build a new path from the about making a difference in his community. “I love main road to his school thanks to generous donations education and helping young kids. T hat’s the reason I from his supporters. keep working in this role,” he shared. “What makes me so proud,” Mr. Pov sh ared, “is that I Mr. Pov enjoyed getting to use the TnT platform to have been granted a huge donation from the Prime submit his textbook requests this year. “The TnT system Minister to build 6 concrete classrooms, 1 big office, and is very useful. I like that I receive a message that shows a school fence. ” the calculated total. Then I can check to make sure the data is correct. If there is a mistake, it is easy to correct Mr. Pov is very excited to continue making it and resend the data,” he explained. improvements to his school. We are glad that the TnT system can contribute to this effort. “The old paper-method takes too long,” he continued. “This new method is faster and easier,” explained Mr. Pov. “I also like that this allows me to double -check the data that I enter. I think that is very helpful,” he said. Mr. Pov requesting textbooks through the TnT system 43 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Annex 3: Template for Recording Book Request Data Grade Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Book Request Data 6 Expected Grade 1 2018-2019 Expected Grade 1 2019-2020 Student Numbers beginning of the year Drop out number Repetition number Number transfer in usable book of Khmer usable book of Social usable book of Math usable book of English usable book of science Record number of books requested by TnT (from telegram message) requested number of Khmer books requested number of Social requested number of Math requested number of English requested number of science 44 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Annex 4: Summary of Findings from the Baseline in 2017 Summary of Key Findings TnT Design Response Requesting School book request processes and forms TnT will calculate book projections reducing use complicated formula that are open to calculation errors error. School book requests are delayed in reaching Requests will be immediately generated after the the Department for Curriculum Development school reports and visible to District, Province (DCD) as paper forms are sometimes and DCD. District and Province will have submitted late and then they must travel to deadlines for approvals preventing delays. district, province and then DCD. The DCD inputs and manages all book DCD will have book requests immediately and projections, budgets and distribution lists in can then use for next year’s budget rather than Excel spreadsheets which results in long delays two years in advance. and errors. DCD makes cuts to school book requests after TnT will provide criteria for DCD to adjust budget approval in an unstandardized, requests fairly based on budget and the system subjective way and it is difficult to track will track all changes made providing adjustments. transparency. Procurement Procurement takes just under a year and is Procurement processes and Ministry of Economy delayed by the fiscal year start date. and Finance (MEF) timeframes regarding fiscal Contracts are not signed until September – year start and end and budget approvals are 2 months before the start of the school year. beyond the sphere of influence of this project. This is an unrealistic timeframe to print and However, by making clearly visible the reality of distribute all textbooks to all schools, hence book delivery dates due to procurement delaying deliveries. bottlenecks, the project may bring extra impetus to procurement reforms. Distribution Books are not packed per school. They are TnT will require districts to record packed and delivered by district and the electronically what is allocated to schools and district is expected to re-pack and allocate this will be confirmed by the school directors to school directors. This opens the supply providing visibility into this link in the supply chain to significant vulnerabilities. chain. The Publishing and Distribution House This issue is related to both the delays in (PDH) is not able to store, pack and deliver contracting and the size of the PDH warehouse. all books in one round before school starts. Both of these issues are beyond the sphere of Each year there are at least two deliveries influence of this project. However, the project to each district, if not three – final deliveries will be able to provide better visibility into the end 3 months after school starts. This was extent of delays, potentially leading to reforms. confirmed by our research findings; school directors from remote/rural schools reported even later deliveries. The PDH is poorly equipped in terms of PDH has committed to upgrading their technology. Picking, packing and labeling is technology hardware, installing internet and done manually. DCD’s national textbook using TnT for tracking deliveries and printing distribution lists are manually re-entered receipts. into an outdated computer program in order to print receipts (using old printers). Bar coding for books and boxes will not be There is currently no Internet. possible, as it will require training and 45 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Summary of Key Findings TnT Design Response equipping many short-term laborers. Instead, picking lists and receipts will be barcoded and warehouse managers will use TnT to log deliveries and record the number of boxes per delivery. The PDH sub-contracts DOE ‘agents’ for This will be a topic in the trainings but issues final book allocation and distribution to of payment of transport are outside this schools. It is generally accepted that school project’s scope. directors must collect books from the DOE. Nearly all school directors reported they collect textbooks from the DOE using own funds for transport. According to PDH, DOE agents are provided with funds to cover transportation of books from the DOE to schools. Teachers feel strongly that MoEYS should The TnT dashboard will provide good visibility take a stronger role in overseeing and into the book supply monitoring textbook supply. SSCs emphasize that they should have a role SSC will be engaged to confirm receipt of books in the textbook supply chain, particularly at at the school and will conduct spot checks of the delivery end, for checking correct classrooms delivery and distribution to students. Results Based Financing of Printers/Distributors Contractors 100% of the contract value is paid to the There has been overall agreement that TnT contracted Private Printing House before data could be used to help determine the point delivery as funds must be spent by the end at which the deposit can be released. of the calendar year. However Private Printing House has to pay 10% of the value of the contract into a bank account as a deposit. This deposit is released in between February and April after spot checks of a sample of receipts have occurred. According to the Procurement Unit, the deposit can only be released when any found issues have been resolved. Results Based Incentives for Schools There is unanimous agreement that School Agreement has been given to use POE budget Support Committees (SSCs) (with additional to provide a certificate/plaque of official helpers such as teachers, grade 6 students acknowledgement to SSCs that fulfill TnT etc.) should take the key role in checking engagement criteria. textbook delivery and use in the classroom. FGDs indicated that being given a role in textbook monitoring would be an incentive in itself and receiving public recognition emerged as another motivational factor. 46 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Annex 5: Data Tables Table 1: Percentage Who Responded That All Students Have All Textbooks (by school category) (NEP) Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 BL EL BL EL BL EL BL EL BL EL BL EL Category Subject Teacher Teacher Teacher Teacher Teacher Teacher Teacher Teacher Teacher Teacher Teacher Teacher Student Student Student Student Student Student Student Student Student Student Student Student Groups Groups Groups Groups Groups Groups Group Group Group Group Group Group Khmer 54% 21% 57% 25% 18% 8% 33% 0% 7% 0% 47% 0% 25% 7% 36% 0% 0% 6% 14% 0% 0% 8% 0% 0% Math 8% 14% 64% 50% 0% 8% 20% 0% 7% 0% 13% 0% 13% 7% 14% 0% 0% 0% 0% 0% 0% 8% 0% 0% 1 Sciences 0% 0% 7% 17% 18% 23% 13% 0% 7% 7% 13% 0% 19% 0% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% Social 25% 7% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% English 56% 93% 14% 10% 8% 65% 14% 27% 0% 54% 20% 31% Khmer 14% 0% 18% 11% 20% 0% 0% 0% 0% 0% 0% 0% 20% 0% 0% 0% 17% 0% 13% 0% 0% 0% 14% 13% Math 0% 0% 0% 0% 10% 38% 20% 9% 14% 10% 0% 0% 20% 0% 13% 0% 17% 0% 0% 0% 0% 0% 0% 13% 2 Sciences 14% 11% 0% 0% 0% 25% 10% 0% 0% 0% 8% 0% 0% 0% 13% 10% 0% 0% 0% 14% 0% 0% 0% 13% Social 0% 0% 0% 0% 0% 0% 0% 14% 0% 0% 0% 0% English 20% 100% 13% 20% 17% 50% 13% 29% 0% 100% 0% 13% Khmer 13% 19% 21% 7% 23% 3% 19% 6% 25% 9% 13% 4% 7% 11% 8% 0% 8% 11% 7% 0% 15% 15% 5% 0% Math 6% 3% 14% 14% 3% 3% 4% 0% 13% 0% 8% 4% 10% 11% 8% 0% 12% 0% 4% 0% 11% 15% 0% 0% 3 Sciences 9% 16% 7% 10% 10% 6% 11% 3% 19% 6% 8% 4% 7% 0% 0% 0% 8% 0% 0% 4% 11% 0% 5% 0% Social 10% 9% 0% 7% 8% 0% 4% 0% 7% 4% 0% 0% English 52% 60% 27% 32% 35% 59% 18% 35% 56% 52% 19% 29% 47 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table 2: % of Teachers and Students Who Said Students Received Textbooks (by grade and subject at baseline and endline) (NEP) Teachers Students Baseline Endline Baseline Endline Khmer 1 73% 68% 87% 86% 2 80% 83% 96% 98% 3 89% 80% 97% 99% 4 83% 85% 94% 100% 5 92% 89% 94% 100% 6 95% 94% 92% 96% Math 1 95% 74% 94% 79% 2 96% 85% 84% 97% 3 89% 93% 97% 99% 4 86% 88% 94% 100% 5 90% 99% 100% 100% 6 96% 100% 92% 96% Science 1 92% 95% 91% 91% 2 91% 89% 82% 99% 3 91% 90% 96% 99% 4 91% 93% 100% 97% 5 97% 100% 100% 94% 6 96% 98% 100% 96% Social 4 88% 98% 95% 98% 5 97% 99% 100% 95% 6 98% 100% 99% 100% English 4 57% 82% 16% 79% 5 80% 85% 42% 70% 6 81% 87% 31% 76% 48 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table 3: Responses to the Question: “Did students receive textbooks from school on time (during November)?” (NEP) Yes, received on time Category Interviewee BL EL Student Groups 90% 70% 1 Teachers 86% 53% School Director 55% 0% Student Groups 93% 75% 2 Teachers 92% 59% School Director 38% 38% Student Groups 85% 66% 3 Teachers 94% 50% School Director 71% 0% Student Groups 89% 70% Average Teachers 91% 54% School Director 55% 13% 49 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table 4: Responses Regarding Which Months Books Were Received Among Those Who Stated That Books Were Not Received On Time (NEP) December January February March April or later Category Interviewee BL EL BL EL BL EL BL EL BL EL School Director 18% 6% 64% 44% 9% 11% 9% 0% 0% 39% 1 Teachers 20% 8% 60% 24% 0% 37% 20% 6% 0% 25% School Director 33% 0% 67% 30% 0% 20% 0% 0% 0% 50% 2 Teachers 20% 3% 60% 15% 20% 30% 0% 18% 20% 33% School Director 32% 3% 37% 33% 5% 15% 11% 3% 21% 48% 3 Teachers 50% 11% 33% 25% 67% 32% 0% 13% 33% 19% 50 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table 5: Percent Received vs Approved (Salesforce) Book Name ↑ Sum of Books Received by SD Sum of Previous Year Books Approved Percent Received vs Approved Record Count ា ក់ទី1 a ភាសាខ្មែ រ ថ្ន 10649 11071 96% 415 b គណិត វ ិទា ថ្ន ា ក់ទី1 5225 5372 97% 415 c វវ.ិ សង្គ ម ថ្ន ា ក់ទី1 5206 5257 99% 410 ា ក់ទី2 d ភាសាខ្មែ រ ថ្ន 13864 14007 99% 415 e គណិត វ ិទា ថ្ន ា ក់ទី2 8250 8377 98% 416 f វវ.ិ សង្គ ម ថ្ន ា ក់ទី2 8541 8620 99% 416 ា ក់ទី3 g ភាសាខ្មែ រ ថ្ន 7783 8150 95% 239 h គណិត វ ិទា ថ្ន ា ក់ទី3 10259 10259 100% 410 i វវ.ិ សង្គ ម ថ្ន ា ក់ទី3 10140 10101 100% 407 ា ក់ទី4 j ភាសាខ្មែ រ ថ្ន 4831 4869 99% 415 k គណិត វ ិទា ថ្ន ា ក់ទី4 4993 5012 100% 415 ិ l វវទាសាដសរ ា ក់ទី4 ថ្ន 5000 5023 100% 415 ា ក់ទី4 m សិកាសង្គ ម ថ្ន 4911 4895 100% 414 ា ក់ទី4 n ភាសាអង្់គ្គេ ស ថ្ន 5042 5254 96% 251 ា ក់ទី5 o ភាសាខ្មែ រ ថ្ន 6453 6504 99% 410 p គណិត វ ិទា ថ្ន ា ក់ទី5 6620 6666 99% 411 ិ q វវទាសាដសរ ា ក់ទី5 ថ្ន 6505 6550 99% 411 ា ក់ទី5 r សិកាសង្គ ម ថ្ន 6556 6554 100% 411 ា ក់ទី5 s ភាសាអង្់គ្គេ ស ថ្ន 8021 8170 98% 409 ា ក់ទី6 t ភាសាខ្មែ រ ថ្ន 6554 6547 100% 415 u គណិត វ ិទា ថ្ន ា ក់ទី6 6609 6664 99% 415 ិ v វវទាសាដសរ ា ក់ទី6 ថ្ន 6555 6608 99% 415 ា ក់ទី6 w សិកាសង្គ ម ថ្ន 6506 6547 99% 415 ា ក់ទី6 x ភាសាអង្់គ្គេ ស ថ្ន 9090 9652 94% 418 Total 174163 176729 99% 9583 51 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table 6: Spot Check Results (Salesforce) Sum of Students with Books Sum of Students Checked % Student with book Kampong Trabek Serei Sophoan Kanhchriech Preah Sdach Phnom Srok Svay Check Peam Ro Pearaing Ochrove Malay Subject ↑ Grade ↑ Khmer 1 93% 89% 92% 87% 87% 94% 79% 80% 79% 83% 7847 6776 86% 2 98% 92% 91% 89% 90% 92% 82% 96% 89% 88% 5195 4709 91% 3 93% 90% 96% 105% 70% 82% 85% 76% 86% 89% 4864 4339 89% 4 83% 97% 86% 93% 159% 85% 97% 60% 94% 81% 3976 3852 97% 5 94% 97% 97% 89% 95% 89% 100% 67% 94% 88% 2616 2434 93% 6 97% 92% 99% 96% 98% 100% 95% 84% 94% 94% 3266 3074 94% Subtotal 93% 92% 93% 92% 97% 90% 83% 78% 87% 87% 27764 25184 91% Math 1 88% 93% 100% 81% 72% 69% 61% 89% 74% 59% 2465 2076 84% 2 99% 96% 98% 90% 89% 90% 88% 72% 116% 90% 3810 3496 92% 3 94% 91% 94% 102% 89% 93% 81% 82% 87% 83% 3445 3122 91% 4 96% 95% 99% 96% 95% 100% 97% 68% 92% 100% 3222 3030 94% 5 97% 95% 96% 91% 95% 100% 80% 94% 100% 2627 2464 94% 6 97% 90% 99% 96% 99% 99% 87% 98% 89% 98% 3023 2842 94% Subtotal 96% 93% 98% 93% 90% 92% 84% 78% 96% 87% 18592 17030 92% Social 1 88% 100% 64% 33% 66% 71% 1547 1241 80% 2 97% 94% 98% 87% 100% 85% 81% 83% 80% 1815 1645 91% 3 99% 88% 97% 107% 91% 79% 89% 92% 150% 100% 2526 2382 94% 52 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 4 100% 95% 97% 96% 96% 77% 97% 88% 80% 95% 1893 1778 94% 5 100% 97% 100% 94% 96% 100% 89% 84% 96% 1737 1651 95% 6 99% 92% 94% 101% 78% 100% 87% 91% 90% 93% 3020 2757 91% Subtotal 99% 92% 96% 94% 86% 86% 83% 86% 113% 93% 12538 11454 91% Science 4 106% 97% 100% 92% 93% 100% 65% 1440 1368 95% 5 98% 98% 94% 100% 100% 100% 90% 100% 1403 1366 97% 6 98% 89% 100% 100% 85% 100% 93% 95% 1634 1502 92% Subtotal 100% 95% 97% 98% 89% 100% 100% 85% 97% 4477 4236 95% English 4 98% 96% 100% 85% 91% 66% 95% 1323 1224 93% 5 100% 96% 100% 132% 100% 103% 1104 1107 100% 6 100% 92% 100% 79% 100% 86% 1117 999 89% Subtotal 100% 95% 100% 89% 95% 100% 93% 95% 86% 3544 3330 94% Total 95% 93% 95% 93% 92% 90% 84% 81% 92% 88% 66915 61234 92% 53 Enhancing Book Distribution in Cambodia World Education, Inc. Contract: 7184295 Table 7: % of Books Received Compared to Books Approved (by grade and subject) Book Name↑ Sum of Books Received by SD Sum of Previous Year Books Approved Percent Received vs Approved Record Count Khmer 1 11256 11472 98% 415 Math 1 5297 5372 99% 415 Social 1 5228 5257 99% 410 Khmer 2 13874 14007 99% 415 Math 2 8252 8295 99% 416 Social 2 8571 8620 99% 416 Khmer 3 8397 8570 98% 239 Math 3 10282 10304 100% 410 Social 3 10156 10129 100% 407 Khmer 4 4819 4869 99% 415 Math 4 4967 5012 99% 415 Science 4 4970 5023 99% 415 Social 4 4875 4895 100% 414 English 4 5194 5262 99% 251 Khmer 5 6492 6504 100% 410 Math 5 6645 6666 100% 411 Science 5 6514 6561 99% 411 Social 5 6583 6554 100% 411 English 5 8072 8179 99% 409 Khmer 6 6532 6569 99% 415 Math 6 6594 6682 99% 415 Science 6 6509 6609 98% 415 Social 6 6492 6548 99% 415 English 6 9658 9697 100% 418 Total 176229 177656 99% 9583 54