Policy Research Working Paper 9689 Lowering Prices of Pharmaceuticals, Medical Supplies, and Equipment Insights from Big Data for Better Procurement Strategies in Latin America Mihály Fazekas Alexandre Borges de Oliveira Nóra Regös Latin America and the Caribbean Region June 2021 Policy Research Working Paper 9689 Abstract Containing rapidly growing health care costs in the Latin forests) methods are used to explain prices as a function of American and the Caribbean region, especially amid the procurement decisions and outputs, such as the number of COVID-19 pandemic, requires an in-depth analysis of prices bidders. Based on in-depth discussions with policy makers, from a novel perspective. This paper documents hitherto the paper also devises realistic policy interventions, which understudied variations in prices paid for pharmaceuticals, in turn can be used to estimate savings scenarios. First, the equipment, and medical supplies within countries and mar- findings show that the prices paid vary greatly across and kets. It also identifies effective procurement strategies for within countries. The latter is surprising given that the reg- lowering prices within existing regulatory frameworks. The ulatory and institutional framework is largely fixed within analysis uses public procurement data gathered by govern- each country. Second, a high proportion of within-country ments’ electronic procurement systems in nine countries and -market variation can be explained by standard features and territories across the region. The data are uniquely of procurement policy implementation, such as the length detailed and complete, encompassing the minute detail of of advertising tenders. Third, the explanatory models point purchasing decisions and processes made across all regulated to the potential for lowering prices across the region by public entities in the study countries and territories. Tra- about 14 percent by implementing low-level, yet impactful ditional regression analysis and machine learning (random changes to how purchasing is done. This paper is a product of Latin America and the Caribbean Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at aoliveira@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Lowering Prices of Pharmaceuticals, Medical Supplies, and Equipment: Insights from Big Data for Better Procurement Strategies in Latin America Authors: Mihály Fazekas (Central European University and Government Transparency Institute), Alexandre Borges de Oliveira (World Bank), Nóra Regös (Government Transparency Institute) JEL classification: C53, D44, H51,H57, I11, C57, C55, C61 Keywords: auctions, Latin America, procurement, pharmaceuticals 1. Introduction The COVID-19 pandemic has put a lot of pressure on already strained health care systems across the Latin America and the Caribbean (LAC) region (World Bank, 2020). Among others, the explosion of costs, such as prices paid for pharmaceuticals (Chernew & May, 2011), has contributed to failings in equitable and quality health care provision in the region. COVID-19 emergency states and the accompanying pressures to deliver critical goods and services fast have further contributed to price pressures in already fragmented markets. Responding to these policy challenges, this study aims to ● document hitherto understudied variations in prices paid for pharmaceuticals, equipment and medical supplies within countries and markets; and ● identify effective procurement strategies for lowering prices within existing regulatory frameworks with the help of purchase-level explanatory models. These ambitions are made possible by the unique data set compiled from public sources enabling price analysis at the very detailed, purchase level (i.e. within countries and markets over time). This represents a major departure from most prior studies which had to rely on market-level average prices across countries (e.g. Danzon & Chao, 2000). We make use of public procurement data as gathered by public authorities in 9 countries and territories across the LAC region through their electronic procurement systems. This data is uniquely detailed and complete, encompassing the minute detail of purchasing decisions and processes made across all regulated public entities in the study countries and territories. Our sample of 7 countries (Ecuador, Brazil (federal), Paraguay, Panama, Uruguay, Peru, and Costa Rica) and 2 territories (Amazonas (Brazil) and Santa Catarina (Brazil)) is exceptional in its breadth; still, it misses out on important countries in the region and hence may represent a skewed sample. Nevertheless, the wide country coverage of our sample allows for making cross-country claims in the region, by exploiting the variance in terms of institutional and regulatory set-up. We make use of traditional regression analysis as well as machine learning (random forests) methods to explain prices paid for pharmaceuticals, equipment and medical supplies. Based on in-depth discussions with policy makers, we devise realistic policy interventions which in turn can be used to estimate savings scenarios. First, we find that prices paid even for highly standardized pharmaceuticals vary greatly across as well as within countries. The latter is surprising given that regulatory and institutional frameworks are largely fixed within each country. Second, a high proportion of within-country and within-market variation can be explained by standard features of procurement policy implementation such as the length of advertising tenders. Third, our explanatory models point at the potential for lowering prices across the region by about 14% by implementing low-level, yet impactful changes to how purchasing is done. 2 2. Institutional set-up: Health sectors, procurement rules and purchasing discretion Latin America and the Caribbean countries’ economies have been weakened by multiple adverse events throughout the past decades, such as extreme drop of commodity prices, rigorous financial constraints, and natural disasters (World Bank, 2020). These events also had a negative impact on social care systems, resulting in lower quality, unequal health care services. From 1990 onwards, countries in the region put great effort in changing the situation and introduced several health care reforms to reduce inefficiency, inequity and increase coverage of health systems, however there is still room for improvement (OECD, 2020). During the ‘Golden Decade’, between 2003 and 2013 (except the 2008 financial crisis), the region experienced stable economic growth, which led to higher public spending in the health care sector, although in the past 5-6 years a declining tendency is observable, contributing to political unrest in many countries in the region in 2019 (World Bank, 2020). In addition to the already nonoptimal conditions, COVID-19 further burdens these countries’ health care systems and their economies (World Bank, 2020). Weaker health systems and health services coverage are also indicated by the high level of out-of-pocket health expenditures, accounting for 34% on average of total health spending (OECD average is 21%) (OECD, 2020). Public health expenditure is low (3.8% of GDP) in the LAC region compared to OECD countries (6.6% of GDP). Public spending, the share of total health expenditure and compulsory insurance is 54.3%, while the OECD average is 73.6%, which deepens unequal access to health care services. Provision of higher quality health care services could also be achieved by more efficient spending (OECD, 2020). Low performance is also compounded by high levels of corruption, for example bribery rates in public health centers are around 11% in the region (OECD, 2020). The market for pharmaceuticals is very heterogeneous with simple, low cost products, such as aspirins, and complex, high cost products, such as cancer therapeutics. Furthermore, some products are purchased in very large quantities while others in small quantities. It can be argued that there are many different sub-markets within the market for pharmaceuticals. In fact, the data set built for this work encompasses 235 products with price per unit varying from USD 9X10 -7 to USD 7,008. This diversity of products requires governments to develop an array of strategies and approaches to procurement if they are to maximize value for money and improve results in this complex market. Our analysis revealed that governments have different strategies and approaches to procurement of pharmaceuticals, leading to varying outcomes. Procurement regulations in place in each jurisdiction as well as choices made by government officials in charge of implementing procurement activities are the two key factors driving results. Procurement regulations and the institutional framework vary substantially across the jurisdictions included in this analysis, driving procurement activities and outcomes. Among these, the most notable are: 3 1. the existence of a centralized procurement agency or centralized approach to procurement of pharmaceuticals, 2. setup of umbrella or framework agreements, 3. requirements for use of competitive versus non-competitive procurement procedures - usually implemented through monetary thresholds. The diversity of approaches within these three parameters were also significant: even when a centralized procurement agency or centralized procurement approach exists, some jurisdictions mandate their use to all health care delivery units (e.g. hospitals or local clinics) while other jurisdictions made these optional. Umbrella or framework agreements were used by some jurisdictions but not by others; and, among those that used umbrella contracts, there are significant differences in terms of mandatory use, number of supply choices in the umbrella contract, institutional arrangements to set up the umbrella agreement and so on. As for competitive, restricted or non-competitive procurement arrangements, the jurisdictions that are part of this work had material differences in terms of the monetary thresholds which are typically used to determine the use of competitive versus restricted or non-competitive procurement approaches. Furthermore, procurement regulations in all jurisdictions had numerous legal avenues allowing for non-competitive procurement even for larger purchases which would require competitive procurement based on monetary value. Finally, some jurisdictions favored the use of reverse auctions for competitive procurement while others relied solely on the traditional tender approach of requesting one sealed bid with the lowest price. In addition to regulatory elements, government officials have considerable discretion to procurement strategies and make implementation decisions, even if they are constrained by national and local procurement regulations. Many of these choices reflect incentives as well as capacity of government agencies. Amongst the key elements for which government officials could make choices that impacted results were: 1. bundling of pharmaceuticals of similar nature as opposed to individual purchases for each item, 2. quantity requested in each purchase, even though this decision can be affected by budgetary allocations and planning procedures, 3. choice of procurement approach in jurisdictions where there is more than one method for competitive procurement (such as electronic versus face-to-face reverse auction, or reverse auction versus traditional bidding and many others), 4. time allowed for bidders to submit a bid, although the minimum time is always defined by regulations, 5. time allocated to evaluate and select the winner, 6. choice to purchase from a framework agreement or carry out a separate individual purchase in jurisdictions where such choice is allowed. Also noteworthy are decisions and procedures outside the scope of procurement processes which had an impact on outcomes, such as: budget and planning procedures, payment terms and turnaround time to release it, court decisions mandating purchase of a certain brand of drug, clinical choices made for a specific brand or for purchase of generic drugs. 4 Variation in the organization of purchasing in terms of which agencies can buy independently, under which conditions, provided the variability of prices as well as procurement strategies to be exploited by the analysis. The key institutional differences are discussed below to offer insights into both the variations we see and the potential scope for policy change: ● Most jurisdictions had a piecemeal approach to procurement of pharmaceuticals with budget allocation specific to each government agency and autonomy to implement the budget. Consequently, procurement of pharmaceuticals was totally decentralized, irrespective of the type of product, resulting in multiple different tenders for the very same product from different government agencies. For agencies with larger budgets, it was very common to find multiple tenders within the same budget year for the very same product. This hands-off approach to procurement strategy resulted in substantial dispersion in prices for the very same product within a certain jurisdiction. This was the prevailing approach in most jurisdictions. ● One national and two subnational governments used procurement strategies to leverage the overall demand of the government to approach markets for supply of high value, complex pharmaceuticals as well as high volume, low complexity ones. These jurisdictions mandated the use of a framework contract to fulfill all demand for the products available within these contracts. A government agency would have to provide written justification if willing to purchase outside the umbrella contract, which was put out for bidding on a yearly basis. This approach resulted in smaller dispersion or, in the case of one of the subnational jurisdictions, no variation in prices paid for many of the products available in the yearly umbrella contract. ● One of the subnational jurisdictions had yearly framework contracts for many products as described above but allowed government agencies to purchase directly if the total award value was below a certain value threshold. Data showed varying uptake of the framework contract across government agencies and, in some cases, multiple small value purchases of the same product even by the same agency. ● One jurisdiction had a fragmented institutional setup for delivery of health services, in which government agencies were autonomous to purchase the products they need. The largest provider had one approach to procurement of pharmaceuticals while the other, much smaller providers followed a different strategy, in most cases purchasing without competition as their scale was small, and individual purchases rarely broke the threshold required for competitive procedures. ● A group of jurisdictions used reverse auctions, either electronically or face-to-face, to purchase the large majority of their demand for pharmaceuticals. ● One jurisdiction had procurement strategies affected by judicial orders that mandated procurement of a product from a specific manufacturer. This had a material impact on prices paid by the jurisdiction as the supplier benefited by the judicial order had almost full leverage over price. Most judicial orders required the government to fulfill the order in a matter of days, making it hard for the government even to try to negotiate down the initial price. These judicial orders required purchase of branded drugs in lieu of generic or 5 cheaper alternatives, and on many occasions came with clinical vouching from a health care provider. 3. Theoretical framework: Explaining unit prices at contract award The determinants of pharmaceutical, medical supplies, and equipment prices have been studied in a wide variety of contexts and methodologies. For example, pharmaceutical prices have been found to be determined by patent expiration and generic producers’ market entry (Morton & Kyle, 2011), production costs (Berndt, 2002), a host of regulatory instruments such as international reference pricing (Wouters & Kanavos, 2017), or quality features (Danzon & Chao, 2000). Markets for medical devices or machinery have also been studied, albeit to a much lesser degree, for example by looking at market structure or bargaining power of market participants (Grennan, 2013). However, there have been relatively few theoretical or empirical investigations of purchasing decisions by public bodies vis a vis suppliers of pharmaceuticals, medical supplies and equipment. A recent notable such study looks at pricing transparency in the public procurement of selected drugs in 2 Brazilian states, Paraiba and Sao Paulo (Kohler et al, 2015). Our theoretical framework takes note of this large body of evidence and looks at the least studied aspect of pricing pharmaceuticals, medical supplies and equipment: public purchases. Such an analytical focus also allows us to bring in and build on a hitherto disconnected literature on the performance of public procurement systems (de Oliveira et al, 2019; Kohler & Dimancesco, 2020). This literature has generated evidence on a diversity of determinants of prices at the point of contract award, some of which overlap with the public health literature briefly enumerated above: the price impact of bidder number, market structure, or centralization of purchasing (Petersen et al, 2020). While there is a host of price predictors largely neglected by the public health literature such as procedure type choice (e.g. open versus negotiated procedure), auction design choices (e.g. length of advertising the tender), or institutional quality (Baldi & Vannoni, 2017; Coviello & Mariniello, 2014). Studying specifically the purchasing decisions made by a wide variety of public buyers across many countries, but within a relatively short time frame (the bulk of the data comes from 2014- 2018), implies that many of the price determinants such as production costs or patent expiry can be assumed to remain largely unchanged from the perspective of our purchase level analysis. Holding such (for us) unobserved factors constant allows the analysis to concentrate on those observed factors which take center stage in public auctions and other types of tenders for standardized health care inputs. These observed price determinants such as market structure, auction design, or bidder features have been derived from at least one of the disciplines we build on: public health, health economics and public procurement. In addition, we also briefly draw on public administration research regarding the effect of administrative quality on prices. 6 The here outlined theoretical framework is derived from the goals of the study, hence it narrowly concentrates on the public procurement decisions and outcomes in health care, while being encompassing in that it covers a wide variety of procurement variables. Such narrow but comprehensive focus is suitable for our research goals because the underlying data is both of a very broad scope virtually including all health care-related regulated public tenders and of great depth capturing detailed information on tendering procedures, the actors and results. The broadest outcome variable we aim to explain is value for money which is defined as the quality of goods or works obtained for a given procurement price. As reliable and comparable data on product quality is hard to obtain, our key focus falls on unit prices at the contract award for (largely) standardized goods. Hence, we define this main dependent variable of interest as Unit price at contract award = total value of items contracted / standardized quantity of items contracted This definition of the dependent variable requires the units of measurement to be standard, at least within markets. This is because we can use market IDs as control variables in the regressions or features in the random forest models, hence making comparisons across purchases within markets rather than across markets. Naturally, units of measurement can vary from market to market (e.g. mg of paracetamol and liters for water). Nevertheless, price elasticities expressed in percent changes can still be compared across markets using our approach. Using unit prices at contract award only approximates, and arguably imperfectly so, value for money. First, the focus on standardized goods and making comparisons with respect to market averages allows for keeping quality differences largely constant; even though some unobserved quality differences may remain. Second, prices at contract award do not take into account cost changes due to contract modification during the implementation phase. Third, payment and delivery schedules (e.g. delivering the same 1000 syringes is likely to have different unit prices if it is to be done by tomorrow versus in 5 months) may differ impacting both the value of provision and production costs injecting an additional unobserved heterogeneity to unit price comparisons. In spite of such shortcomings, there is a wide ranging, policy-relevant, and methodologically diverse scholarship using unit prices in economics, political science, and public health (e.g. Bandiera, Prat, & Valletti, 2009; Collier, Kirchberger, & Söderbrom, 2015; Fazekas & Tóth, 2018; Lewis-Faupel, Neggers, Olken, & Pande, 2016; PricewaterhouseCoopers, 2013; Andrei Yakovlev, Vyglovsky, Demidova, & Bashlyk, 2015). While acknowledging shortcomings, we follow this literature and demonstrate unit prices’ applicability and usefulness for public procurement as well as health economics research. On the explanatory factors’ side of the equation, the conceptual framework aims to be encompassing, capturing all major phases and actors of the purchasing process while also incorporating structural factors (Fazekas & Blum, 2021). Following de Oliveira et al (2019), we 7 group price predictors according to market features and main components of the tendering process such as specifications, actor characteristics (buyers as well as bidders), and outcomes (Table 1). Such simple grouping of factors helps us to develop a comprehensive measurement framework while clearly delineating alternative types of price determinants. In addition, these features are amenable to policy intervention to different degrees facilitating evidence-based policy making in health care. By implication, explanatory factors are gathered into 2 main groups at the end of the analysis: those which are directly determined by policy makers such as purchasing officials (e.g. length of advertising a tender) and those which can only indirectly be influenced by policy (e.g. number of bidders). While it is relatively straightforward to classify each predictor according to this framework, in practice, some of them appear in combinations giving rise to relational features such as physical distance between the buyer and supplier (determining transportation costs among others). 8 Table 1. Summary of main groups of explanatory factors used in the analysis Group Definition Examples Type · Number of potential Market The technological and suppliers Structural characteristics competitive structure of · Technological complexity the market. · Length of advertising Tender The conditions for bids Directly policy specifications bidders to participate in · Conditions of influenceable a tender. participation · Average time taken for Buyer Level of administrative evaluating bids Directly policy characteristics quality of the buying · Buyer office location influenceable office or agency · Company size Bidder/supplier Company productivity · Company headquarters Indirectly policy characteristics and capacity location influenceable · Number of bids Tender Intermediate and final submitted Indirectly policy outcomes results of the tendering · Contract start date influenceable process Source: de Oliveira et al, 2019 Market or product group characteristics encompass those factors which describe the technological and competitive structure of the market in which the items are purchased. Most of these variables are in effect structural givens from the perspective of the analysis as they tend to be changing slowly or policy can only influence them at high cost. Procurement markets can be defined as a combination of product codes and geographical codes (Fazekas & Tóth, 2016) which capture fundamental differences, among others, by product complexity and technological characteristics, geographical characteristics (e.g. remoteness), and market concentration. For 9 standardized goods in health care but also beyond, unit prices are strongly influenced by such factors; for example higher population density increases road costs (100 people by square km leads to a 10% price increase) on a global roads construction sample (Collier et al., 2015), or higher social capital leads to lower goods prices in Italy (Bandiera et al., 2009). Tender specifications capture all the conditions defining who can bid, under which procedural conditions and according to which assessment criteria and award decision rule. These variables are typically directly influenceable by policy makers without changes to the legal framework. Tender specifications variables are defined in the tender preparation and bidding stages of the procurement process. Among many, advertising tenders in a widely used, free, online portal tends to lower prices, such as in Italian public works tenders where the effect size is 7% higher winning rebates (i.e. discounts compared to the reference price) (Coviello & Mariniello, 2014); or longer term, fixed price contracts are more expensive in Russian sugar purchases while larger volumes lead to lower unit prices (Andrey Yakovlev, Bashina, & Demidova, 2014). Buyer characteristics-related variables describe the level of administrative quality or capacity in the buying public entity such as a purchasing office or agency (Fazekas & Czibik, 2021). While many of these characteristics are directly influenceable by policy intervention such as capacity building trainings, many reforms may be challenging to implement such as giving more discretion to purchasing officials in the wake of public demands for accountability and impartiality. Among many others, procurement staff capacity lowers prices across many contexts (Best, Hjort, & Szakonyi, 2017), for example, in the US federal bureaucracy, one standard deviation increase in competence decreases cost overruns by 29% (Decarolis, Giuffrida, Iossa, Mollisi, & Spagnolo, 2020); or increasing procurement officers’ autonomy compared to their auditors reduces prices of standard goods by 9% in Punjab, Pakistan (Bandiera, Best, Khan, & Prat, 2019). Bidder and supplier characteristics capture the key determinants of companies participating in public procurement which determine their ability to offer low prices such as productivity and capacity to deliver to requirements on time. As suppliers are selected through tenders and results at least partially depend on the choices companies make, these characteristics tend to be only indirectly influenceable for policy makers. Among many factors, company location and size are likely to influence prices while company risks such as tax haven registration or political connections also typically impact prices (Fazekas, Tóth, & King, 2016). For example, in India, roads built by politically connected contractors are on average 11% more expensive (Lehne, Shapiro, & Eynde, 2018). Tender outcomes correspond to the intermediate or final results of the tendering process which naturally influence unit prices at contract award. These factors tend to be only indirectly influenceable by policy makers given the important role played by bidder decisions and interactions. A widely cited factor, the number of bidders, plays a highly influential role in determining prices under non-collusive conditions (Fazekas & Kocsis, 2017; Andrey Yakovlev et al., 2014). 10 4. Methodology 4.1 Data Public procurement data underlying the analysis must be high quality, wide scope and high resolution, including information on individual items purchased and their unit prices. We obtained the data sets directly from the official data holders such as national public procurement authorities. Data sets contained structured information on tendering processes such as procedure type used, tendering outcomes such as number of received bids, and information of actors such as buyer and bidder names. More on indicators in section 4.2. Given our goal of looking at health care markets across countries, the biggest challenge of database building was to harmonize data sets coming from different legal and IT systems, for example by matching product classifications from different countries. When compiling the data set for the analysis, we started from the full public procurement data sets in each country and territory: we compiled a data set of 7 countries and 2 territories (2 states of Brazil). As a first step of database creation, we selected all tenders which relate to the health care sector in the source data sets (Table 2). Second, we narrowed down the list of selected health care-related tenders to those which belong to an overlapping product category across all 9 data sets. This harmonization process resulted in a considerable reduction of our database size from about 128,000 tenders to 36,611 tenders, basically due to incommensurate national product classifications at the very detailed level we needed for the unit price analysis. The data was available to the research team largely for the period between 2012 and 2018, with considerable differences across countries. While each data set was defined on the level of item or purchase, in one country data structure differed. Here, the unit of observation was purchase orders within framework agreements which had to be aggregated to unique items within tenders to make them comparable to the other data sets. 11 Table 2. Overview of data used by country and territory Total number of health care- Total number of health care- Years covered related tenders in country related tenders in combined Country data set health care data set Ecuador 22609 22609 2013-2017 Brazil 2014-2016 (federal) 14108 2140 Amazonas 2014-2018 (Brazil) 9030 2797 Santa 2013-2018 Catarina (Brazil) 1348 948 Paraguay 2899 830 2012-2016 Panama 56738 5439 2014-2018 Uruguay 12319 1008 2014-2018 Peru 9217 686 2015 Costa Rica 517 154 2016-2017 Total 128785 36611 Given that the key precondition for unit price analysis is the use of sufficiently detailed and standardized product categories which are in addition comparable across 7 countries, we offer further details on the product codes and their processing. We identified health care-related tenders included in the initial, broad data sets based on product classification codes and higher level descriptions as defined in Annex A, Table A1. The health care-related product code harmonization was based on a host of classification schemes as used in the national data sets following national legislation (Table 3). In the absence of a generic, reference categorization, we considered the Ecuadorian nomenclature as the master reference as it was the most comprehensive for the health care products analyzed in the study. We identified the equivalent classification levels in all data sets by using a semi-automated matching method with extensive manual crosschecks. We excluded unmatched categories from the current analysis to enable cross-country pooled analysis. While a large number of categories and tenders were excluded as 12 a result of this harmonization process, it is not considered problematic as the majority of them referred to chemicals and laboratory equipment which are out of focus of the study1. 1 f or a detailed overview see:https://docs.google.com/spreadsheets/d/1IBxWlGteyrJHOzxovSpWOrIg9K6- 337ZHchIygNYMRg/edit#gid=1534722942 13 Table 3. Product harmonization across countries, health care-related items Total number of Total number of categories in country categories Classification Level of data set (at level of overlapping with Country scheme observation obs) master file (Ecuador) CPC - Clasif icación Central de Level 5 - CPC Level Ecuador Productos 5 222 222 UNSPSC - United Nations Standard Level 4 - Descripción Products and ítem nivel4 Paraguay Services Code (Commodity) 5122 217 UNSPSC - United Nations Standard Products and Level 4 - Nombre Panama Services Code Rubro (Commodity) 5122 217 SICE - Sistema de Inf ormación de Compras y Contrataciones del Estado Level 5 - Descripción Uruguay Artículo 1097 166 CPC - Clasif icación Central de Peru Productos Level 4 - Commodity 1763 203 Catálogo de Brazil Materiais e Serviços (f ederal) Level 3 - Padrao 2445 181 Catálogo de Level 3 - Amazonas Materiais e Serviços Classif ication (Brazil) Commodity 2211 195 Catálogo de Materiais e Serviços Santa Level 3 - Item Catarina Classif ication (Brazil) Description 5996 205 UNSPSC - United Nations Standard Products and Level 5 (f ree text) - Costa Rica Services Code DescProducto 692 92 14 In addition to product classification harmonization, we harmonized unit prices, contract values and procurement methods. Price variables without VAT were converted into international constant USD using official World Bank PPP exchange rates. 2 A host of national procurement methods were grouped into open, restricted, non-competitive and other types. We marked framework agreements separately. 4.2 Indicators When operationalizing the groups of explanatory factors and individual indicators of the theoretical framework we aimed to be as encompassing as possible while considering the limitations of the data set. Hence, we define a large array of indicators and let ‘the data decide’ on which ones are powerful predictors of unit prices (Table 4). Nevertheless, each of these indicators are policy relevant and derive from the literature which is based on tested economic and public administration theory using qualitative as well as quantitative methods. The interpretation based on the relevant literature is discussed in the results section. 2 https://data.worldbank.org/indicator/PA.NUS.PPP 15 Table 4. Summary of variables used in the analysis, LAC region Type Group Variable name Types mean/ most std.div N(non- frequent missing) values* DV - Unit price continuous 8512.9 453443.4 284,872 Structur Market Market ID: ref lecting categorical 287,041 al charact product code (1…235) eristics Year of contract award categorical 2014 287,041 (2012- 2018) Buyer Buyer type (independent categorical National 286,634 charact agency, ministry, etc) gov. eristics Buyer location (region) categorical 287,041 Directly Tender Month of spending categorical July 33,155 policy specific (January, February, etc.) influenc ations eable Procedure type (1 - f ully categorical Fully 279,720 competitive; 2-restricted, competitive etc.) Advertisement period continuous 15.88 17.66 34,923 length (days) Decision period length continuous 29.80 33.07 30,491 (days) Failed tenders (%) continuous 16.01 19.42 22,398 Framework agreement binary Yes 287,041 (Y/N) 16 Product bundling continuous 34.83 38.54 287,041 Quantity of purchased continuous 23198.31 105542.8 287,041 goods (number of units) Indirectl Bidder/ Buyer-supplier f rom the binary No 282,028 y policy supplier same state (Y/N) influenc charact eable eristics Supplier size (micro, small, categorical large 258,341 large company) Supplier specialisation: continuous 83.87 39.65 284,718 number of markets the company supplies Bidding Number of bidders continuous 4.32 7.21 48,703 outcom es Annual winner market continuous 57.04 38.47 284,614 share (%) Annual winner share in continuous 7.01 14.74 284,614 buyer spending (%) Notes: * most frequent values for categorical variables. 4.3 Methods Given the unique scope of the data set we compiled, i.e. item-level health care purchases for 7 countries over many years, we set out to conduct both descriptive and explanatory analyses. The goal of the descriptive analysis is simply to demonstrate the variation in unit prices across as well as within countries. To this end we offer distribution graphs as well as variance decomposition by territory. Given the focus on average prices of most of the previous research, demonstrating the large within country and market variation in prices is both interesting on its own and serves as a motivation for our further analysis. In order to identify effective procurement strategies improving value for money, we also conduct explanatory analyses which incorporate the wide array of indicators listed above. We compare traditional regression methods (Ordinary Least Squares) with random forests in order to identify the model with the best fit. Once the best model is identified, we can derive feasible changes in the predictor parameters, i.e. policy scenarios, which decrease unit prices in the models. 17 Given that the goal of the analysis is to explain variation of prices within each product market across the region, only those markets are considered which have adequate levels of within-market price variability. In our pooled analysis of the whole LAC region, we only retained those markets with at least 100 items awarded worth at least 10,000,000 USD in 2012-2018. The first modeling approach to unit prices brings together all major explanatory factors into a single ordinary least squares (OLS) regression model. Such a comprehensive model allows for system-wide price predictions and simulating hypothetical scenarios. The following linear regression model for log unit prices of standardized products was estimated at the level of item purchased: Log(Pri) = αi + ꞵ1*X1i + ꞵ2 * X2i + ꞵ3 * X3i + εi Where Log(Pri) represents the natural logarithm of unit price for the ith item purchased; X1i stands for the set of directly policy influenceable predictors for the ith item purchased such as the choice of procurement method; while X2i represents the set of indirectly policy influenceable predictors for the ith item purchased such as the number of bidders. X3i denotes the set of control variables accounting for structural factors not amenable to policy intervention for the ith item purchased such as the year of purchase. εi stands for the error term of the regression model. For the sake of simplicity, we opted for a straightforward ordinary least squares estimation of the regression parameters which is a computationally efficient estimator able to handle models run on millions of records while containing hundreds of explanatory variables. Including a battery of fixed effects for structural factors such as market and year allows for building models with exceptionally high explanatory power: across the countries and territories analyzed, R2 ranges between 0.7 and 0.8. In spite of the linear functional form, the modeling framework is able to account for a range of nonlinear relationships by adopting a logged unit price dependent variable and also looking at alternative formulations of the independent variables, for example taking deciles of market concentration as a set of dummies rather than as a continuous scale. Such a simple modeling framework cannot fully account for a range of complex relationships between factors on multiple levels such as market and tender level factors or explanatory factors influencing not only the dependent variable but also each other (e.g. advertising tenders not only influences prices directly but also through the number of bidders). While a full Multilevel Modeling strategy using country, product market and year levels is unfeasible on computational grounds, we systematically explored key interactions based on prior literature (de Oliveira et al, 2019). Hence, we will also report OLS results with such interactions to gauge the improvement in predictive power they confer. The second main models built for explaining unit prices were Random Forests (RF) (James et al, 2015). RF estimation is based on intuitive tree-based models which split the sample in sequences minimizing prediction error. RF is an ensemble method which means that the eventual prediction of an RF model is based on aggregating over a large number of decision trees, each of which is 18 constructed using randomly varying parameters (randomly drawn samples of observations and predictors). Our RF models make use of the same wide array of explanatory variables as OLS models. Given the large number of models we develop, we have to decide on how to choose between them. We consider hard and soft criteria for model choice. The hard criteria for model selection is explanatory power, in particular the mean squared error of the model on an ‘unseen’ test data set (25% random sample of the data). That is testing the prediction accuracy of models on a data set we did not use for developing the models which helps avoiding model overfitting. The soft criteria for model choice, carrying considerably lower weight, is its interpretability and the coefficients’ and partial impacts’ fit with prior literature. The 2 sets of explanatory models, OLS and RF, both carry the advantage of exceptionally wide scope and computational efficiency, even though we cannot claim that the estimated conditional probabilities are causal. Nevertheless, we rely on strong, well-tested theories underpinning the choice of explanatory factors as well as their expected price impacts which together tentatively point at a causal story. In addition, the exceptionally high explanatory power of the models (R2 around 0.7-0.8), unusual for micro-level quantitative models, lends support to the claim that omitted variable bias does not plague our estimations. Finally, once the best model is built, we re-estimate it on the full sample and explore the relationship between individual predictors and unit prices in detail. Upon this exploration and interpretation we devise a set of policy relevant and feasible policy scenarios, that is modifications to the input parameters, which allow for estimating the likely savings to be made from better procurement strategies in health care purchasing across LAC, pre COVID-19 pandemic. 19 5. Results 5.1 A surprising variability of prices Our first goal is to document the variation in prices paid for pharmaceuticals, equipment and medical supplies across the LAC region. The residual variation in unit prices, that is deviation of prices from the regional market average, shows remarkable dispersions within each country (Figure 1). The standard deviation of within market unit prices ranges between 5 USD and 33 USD by country/territory. Naturally, there are some countries/territories where prices are considerably below the regional average, however, crucially for our main argument, most price variation is within market - within country. Overall, cross-country differences account for only 16% of within market price variation with country-year fully interacted factors accounting for only a little under 18% of variation. Figure 1. Histograms of price deviations from market average, by country and territory, log scale 20 To demonstrate these surprising variations, we pulled out a small set of pharmaceuticals which are standardized to a large degree and showed their country averages and dispersions (range between the 25th and 75th percentiles) (Figure 2). Surprisingly, even standard products like ibuprofen are priced very differently across countries and territories (ranging between less than 0.1 USD and 1.5 USD). Nevertheless the 25th percentile of even the most expensive country is below the regional average, suggesting that it is possible to purchase at a low cost even in otherwise expensive local markets. Figure 2. Average unit price by country and territory, selected pharmaceuticals, USD 5.2 Explaining price variation across Latin America After establishing that unit prices for pharmaceuticals, equipment and medical supplies vary a lot within market and country as well as over time, we build simple explanatory models accounting for this variation. First, we start by running simple ordinary-least squares models (OLS) explaining log unit prices in the presence of a host of controls such as market ID, country/territory, and year (Table 5, Model 1). Then, we add directly policy influenceable predictors (Table 5, Model 2) as well as indirectly policy influenceable predictors (Table 5, Model 3). All three of these models are estimated on the train data set, setting aside 25% of the sample for testing prediction accuracy. 21 The controls only model accounts for 77% of total price variation. Interestingly, there is some decrease in average prices between 2012 and 2018 (Table 1, Model 1). Adding directly policy influenceable features lifts explanatory power to 87%, i.e. these factors account for an additional 10% point of price variance (Table 1, Model 2). This model identifies relationships which confirm prior research (De Oliveira et al, 2019). For example, purchasing in larger quantities considerably decreases prices: while the average price is 13 USD in the 5th decile of purchased quantity (on average 862 units bought together), prices drop to 9 USD in the 6th decile (on average 1681 units bought together). Similarly, prices considerably drop in framework agreements: the average prices in framework agreements are predicted to be 21 USD, while 52 outside of it. Adding indirectly policy influenceable factors increases explanatory power to 89%, a small but relevant improvement (Table 1, Model 3). We find a further set of relationships, aligning with prior research. For example, decreasing market concentration is associated with lower prices: when the supplier’s market share drops from the highest decile (76% market share on average) to the 5th decile (27% market share on average), average prices are predicted to decrease from 89 USD to 59 USD. Similarly, local firms (registered in the same state as the buyer) tend to offer somewhat lower prices, on average 36 USD, compared to non-local firms offering on average 43 USD. 22 Table 5. Simple OLS regression model, training data set, LAC region Model 1.Controls 2.Controls+Direct 3.Controls + Direct + policy Indirect policy Predictors/dependent variable Log item unit price (USD) Avg. Failed tenders, Baseline: 0-67% Avg. Failed tenders=67.5-100% -0.0620*** -0.412*** (0.000) (0.000) Avg. Failed tenders=missing -2.539*** -2.904*** (0.000) (0.000) Avg. Decision period, Baseline: 0-8 days Avg. Decision period=9-10 days 0.0812*** 0.134*** (0.000) (0.000) Avg. Decision period=11-17 days 0.297*** 0.396*** (0.000) (0.000) Avg. Decision period=18-39 days 0.500*** 0.560*** (0.000) (0.000) Avg. Decision period=40-358 days 0.492*** 0.600*** (0.000) (0.000) Avg. Decision period=missing -0.397*** -0.518*** (0.000) (0.000) Advertisement period length, Baseline: 1-7 days Advertisement period=8-12 days 0.0649*** 0.0181 (0.000) (0.135) Advertisement period=13-183 days -0.0957*** -0.0872*** (0.000) (0.000) Advertisement period=missing -0.853*** -0.737*** (0.000) (0.000) Procedure type, Baseline: Fully competitive Restricted competition -0.787*** -0.568*** (0.000) (0.000) Non-competitive -0.546*** -0.429*** (0.000) (0.000) Other -0.281*** -0.506*** (0.000) (0.000) Missing 0.777*** 0.626*** (0.000) (0.000) Framework agreement, Baseline: No Framework agreement=Yes -0.937*** -0.993*** (0.000) (0.000) Month of spending, Baseline: December January -0.0614* 0.110*** (0.032) (0.000) February 0.0738** 0.126*** (0.002) (0.000) March -0.150*** -0.0723*** (0.000) (0.000) April -0.0287 0.0581** (0.183) (0.004) May -0.258*** -0.115*** (0.000) (0.000) June -0.247*** -0.0637*** (0.000) (0.000) July -0.204*** -0.0992*** (0.000) (0.000) August -0.0670*** 0.0463** (0.001) (0.009) September -0.00289 0.0486** (0.880) (0.006) October -0.0197 0.0155 23 (0.291) (0.371) November -0.135*** -0.0719*** (0.000) (0.000) Missing 3.001*** 3.888*** (0.000) (0.000) Product bundling, Baseline: 1-3 items 4-8 items -0.550*** -0.512*** (0.000) (0.000) 9-14 items -0.460*** -0.374*** (0.000) (0.000) 15-22 items -0.575*** -0.479*** (0.000) (0.000) 23-27 items -0.400*** -0.297*** (0.000) (0.000) 28-35 items -0.361*** -0.307*** (0.000) (0.000) 36-38 items -0.400*** -0.382*** (0.000) (0.000) 39-50 items -0.599*** -0.420*** (0.000) (0.000) 51-111 items -0.386*** -0.431*** (0.000) (0.000) 112-154 items -0.331*** -0.343*** (0.000) (0.000) Quantity of purchased goods, Baseline: 1-24 Purchased quantity=25-200 -2.629*** -2.661*** (0.000) (0.000) Purchased quantity=201-582 -3.497*** -3.491*** (0.000) (0.000) Purchased quantity=583-3000 -4.130*** -4.073*** (0.000) (0.000) Purchased quantity=3001-10000 -4.363*** -4.397*** (0.000) (0.000) Purchased quantity=10001-39110 -4.696*** -4.719*** (0.000) (0.000) Purchased quantity=39111-100000 -5.175*** -5.156*** (0.000) (0.000) Purchased quantity=100001-319146 -5.515*** -5.543*** (0.000) (0.000) Purchased quantity=319147-1000000 -5.996*** -6.065*** (0.000) (0.000) Purchased quantity>1000000 -6.760*** -6.904*** (0.000) (0.000) Supplier specialisation, Baseline: 1-8 markets Supplier is on 9-14 markets -0.0291** (0.009) Supplier is on 15-52 markets -0.367*** (0.000) Supplier is on 53-69 markets -0.671*** (0.000) Supplier is on 70-86 markets -0.904*** (0.000) Supplier is on 87-98 markets -0.750*** (0.000) Supplier is on 99-109 markets -1.005*** (0.000) Supplier is on 110-116 markets -1.114*** (0.000) Supplier is on 117-123 markets -1.317*** (0.000) Supplier is on 124-139 markets -1.243*** (0.000) 24 Missing 1.414*** (0.000) Buyer-supplier location, Baseline: No Buyer-supplier location: Yes -0.200*** (0.000) Buyer-supplier location: Missing 0.554*** (0.000) Company size, Baseline: large medium 0.0786*** (0.000) small 0.0719*** (0.000) Missing 0.973*** (0.000) Market concentration (by country), Baseline: 0-6.1% Market concentration=0.04-18% 0.722*** (0.000) Market concentration=0.12-59.4% 1.014*** (0.000) Market concentration=0.26-59.4% 1.067*** (0.000) Market concentration=0.65-84.6% 1.128*** (0.000) Market concentration=1.3-98% 1.091*** (0.000) Market concentration=2.9-100% 1.113*** (0.000) Market concentration=5.9-100% 0.984*** (0.000) Market concentration=12.7-100% 1.387*** (0.000) Market concentration=28.7-100% 1.516*** (0.000) Buyer spending concentration (by country), Baseline: 0-1.2% Buyer spending conc.=0.01-2.5% 0.416*** (0.000) Buyer spending conc.=0.06-4.3% 0.574*** (0.000) Buyer spending conc.=0.2-7.7% 0.747*** (0.000) Buyer spending conc.=0.5-12.7% 0.855*** (0.000) Buyer spending conc.=0.9-24.6% 1.023*** (0.000) Buyer spending conc.=1.7-53.5% 1.200*** (0.000) Buyer spending conc.=2.5-89% 1.352*** (0.000) Buyer spending conc.=4.9-100% 1.637*** (0.000) Buyer spending conc.=10-100% 1.723*** (0.000) Bidder number, Baseline: 1-2 Bidder number=3-7 -0.197*** (0.000) Bidder number=8-75 0.110*** (0.000) Bidder number=missing -0.389*** (0.000) Constant Y Y Y Buyer type Y Y Y Buyer region Y Y Y 25 Contract award year Y Y Y Market ID Y Y Y Observations 284448 284448 282853 R-squared 0.769 0.868 0.886 p-values in parentheses * p<0.05 ** p<0.01 *** p<0.001 Building on the simple, still comprehensive OLS model, it is possible to refine the analysis by allowing greater room for cross-country differences. The OLS models in Table 5 enforced the same slope or response intensities on all countries even though some variation by country in predictor definitions was allowed (e.g. defining deciles country by country rather than uniformly across all countries). This may not fit the structure of the data in all cases which may cause some relationships to be much weaker than prior research looking at individual countries. Hence, in cases where the initial findings warrant, we will allow for slopes and intercepts to vary country by country (For full details see Annex B). 3 For example, the number of bidders showed an ambiguous impact on prices in the simple OLS models of Table 5. However, when we allow for bidder numbers to have a different impact function across countries, different non-linear effects arise (Figure 3). In some countries/territories as the number of bidders decreases, prices continuously fall, albeit with different magnitudes. While in other countries/territories the decline in prices is particularly concentrated at the lower end of the scale, that is when bidder numbers increase from 1 to 2 or 3 (Fazekas & Kocsis, 2017; Andrey Yakovlev et al., 2014). Interestingly, at the upper end of the distribution, at 4 or 5+ bidders, prices start to increase in some countries/territories. 3 Please note that f or these interacted regressions, we simplified the buyer location control variable from region to country/territory (only 8 categories). This makes the interactions tractable and easily visualizable, albeit R2 decreases somewhat. 26 Figure 3. Margins plot: The impact of bidder number categories on log unit prices, country-by- country effects The month of purchase typically tends to follow a distinct pattern in which the end of the financial year, about the last 1-3 months, tends to be considerably more expensive than the other months (de Oliveira et al, 2019). However, this pattern is not clearly present in our simple OLS regressions (Table 5, Model 2 and 3). However, when each country is allowed to have its own monthly price impacts a variety of patterns emerge (Figure 4). In some cases, the October-December period is clearly the most expensive part of the year. Moreover, many countries/territories show high prices at the beginning of the year, in January-February. By and large, in most countries/territories, prices are lowest in the middle of the year. 27 Figure 4. Margins plot: The impact of purchase month on log unit prices, country-by-country effects The alternative method we apply for predicting log unit prices is a random forest algorithm. Two meta parameters have to be decided for finding the optimal model: number of trees and the number of variables to sample at each run. After systematically searching for the best solution, we opted for 175 trees and 6 variables (for details see Annex C). This model has an explanatory power of 83% which is similarly high compared to the regressions methods discussed above. The advantage of a random forest algorithm is that it is able to incorporate non-linear relationships as well as a host of interactions among predictors. For example, the number of bidders are predicted to have a strong negative effect on unit prices until 8-9 bidders, after which prices rise considerably (Figure 5). However, please note that about 90% of items receive fewer than 10 bids, hence for the bulk of the sample, we observe the expected negative relationship. Also quite informative is the estimated price impact of market share of the supplier with the expected positive relationship throughout (Figure 6). However, the relationship is non-linear with especially steep price increases at the upper end of the distribution around 80%-90% market shares. 28 Figure 5. Partial dependence plot: Number of bids Figure 6. Partial dependence plot: Supplier annual market share (as ratio) 29 As we have developed a range of different models, we have to select the best one for the savings calculations following the criteria outlined in the methods section (prediction accuracy on the test data set and model interpretability). We used the most complete model from each type of model developed, simple OLS, interacted OLS, and Random Forest. Model prediction accuracy is summarized in Table 6 using R-squared and Mean Squared Error metrics. While the results are not unequivocal, the most interpretable model, the simple OLS regression, performs almost as good as the 2 others suggesting that it represents a balanced trade-off between precision and interpretability. Hence, this will be used for the subsequent savings estimations. Table 6. Model comparison table, test data set Model R2 MSE OLS (Table 5, model 3) 0.886 2.612 Interacted OLS (Table B1, Model 3) 0.887 2.401 Random Forest 0.847 1.451 Note: Boosting was also estimated but no significant improvement over Random Forest was found. 5.3 Savings strategies and price impacts Based on the prior advanced statistical models, it is possible to estimate hypothetical unit prices under alternative policy scenarios which reflect how much the government would have paid for the particular product had it used a more cost-effective procurement strategy. The basis of such estimation is the standard OLS selected above which model expresses the associations between various policy relevant factors and unit prices. In addition, we define realistic yet decisive policy changes to be used in the savings estimates. We start by outlining these suggested improvements to procurement policy (Table 7), then we show the estimated impact in the aggregate (Figure 7). The savings model, composing of the price regression and the savings strategies, identifies about 14% savings potential per year, across the studied whole LAC health care sector (Figure 7). Such a substantial total predicted savings is achieved by drawing on only the most readily influenceable factors and assuming only relatively achievable policy changes. None of these changes requires altering the regulatory framework, rather imply relatively small tweaks to procurement process design and implementation decisions. Such a savings potential is roughly in line with prior savings identified looking at individual countries but across a host of economic sectors (de Oliveira et al, 2019). 30 Table 7. Summary of savings strategies used for price predictions directly policy quantity of purchased moving 30% of the lower deciles (582< units bought) to inf luenceable goods the next higher decile (583-3000 items bought) procedure type move 5% of items of competitive procedures to restricted procedure types advertisement period move 20% of shorter advertisement periods (1-12 days) length to longer advertisement (13-183 days) month of spending smooth spending across for Dec, Jan and Febr by reallocating 30% of items to a nearby cheaper month (March) product bundling moving 5% of items in the 1st decile to the 2nd decile f ramework agreement moving 2% of contracts without framework agreement to with f ramework agreement organisational quality: moving 7,5% of tenders in the longest quintiles to the 3rd avg. decision making quintile period length (days) organisational quality: move 50% of items from the lowest half success rate avg. f ailed tenders % organisations (< 67.5%) to highest success rate organisations (>67.5%) indirectly policy bidder number moving 13% weakly competitive items (1-2 bidders) to inf luenceable more competitive items (2-7 bidders) buyer spending move 5% of items from high spending concentration concentration buyers (the top 4 highest deciles of buyers) to average spending concentration buyers (5th decile) market concentration move 50% of items in the highest concentration markets (the top highest decile market) to lower concentration markets (8th highest decile) same location increase market share of local suppliers (i.e. same state) by 10% company size decrease share of small- and medium-sized companies by 20% (to the advantage of large companies) supplier specialisation moving 3% of items supplied by highly specialised suppliers (1-3 lowest deciles) to average specialised suppliers (4th decile) 31 The largest price impact is predicted for bundling products in the same auction, rather than buying them separately (Figure 7). A 2.1% price drop is expected by our model if 5% of items bought in single market actions (i.e. auctions procuring items from 1 market on average) are moved to auctions with 4 markets on average. Most other directly policy influenceable predictors can be used to achieve about 1-1.5% savings such as increasing the quantity bought in one auction from 582< units to 583-3000 units. Among the indirect policy influenceable factors, bidder number and spending concentration have the strongest price impact in our savings model. Increasing competition in only a fraction of tenders (13%) from 1-2 bidders to 2-7 bidders is estimated to lower unit prices by 1.2%. Similarly, lowering individual buyers’ spending concentration from the top half of the distribution (6-10th deciles) to the middle of the distribution (5th decile) is predicted to lower prices by 1.3%. Figure 7. Summary of total savings associated with each intervention, % unit price decrease, LAC 32 6. Conclusions This study has looked at a hitherto understudied aspect of health care pricing in the Latin America and Caribbean region: procurement policies. Our novel analytical focus was enabled by a unique, high-granularity item-level public procurement database covering 7 countries and 2 territories throughout the 2012-2018 period. We analyzed the impact of a host of policy-relevant predictors on unit prices for pharmaceuticals, equipment, and medical supplies, using a range of regression and machine learning methods. Overall, our models perform exceptionally well, accounting for over 80% of price variance in unseen data. These models enabled us to draw on a set of realistic policy changes to produce expected price savings in the health sector across the region. Overall, the identified small-scale interventions are predicted to lower unit prices by about 14%. Our approach is novel, but it has a range of strengths and weaknesses which could be taken up by future research. The strength of the analytical framework rests in its rich empirical basis, government-wide administrative data of great detail; the strong links between well-established economic theories and the relationships sought; and the diversity of analytical methods, both quantitative and qualitative, leading to similar conclusions across the board. Moreover, our experiences with applying the framework in 9 LAC countries and territories suggest that the methods are replicable and largely standard across diverse contexts. Even though regulatory, data, and market differences require a great deal of care and adaptation of the methodology. However, the analytical framework is not without its weaknesses, some of which can be addressed in future iterations or with better data. Most fundamentally, missing and incorrect data can invalidate findings which is, luckily, a relatively limited problem in transactional e-procurement systems. Nevertheless, highly technical and seemingly minor problems can pose substantial challenges. For example, lack of sufficiently standardized units of measurement or dosage information can render unit price comparisons flawed. While the analyzed products are highly standardized across the region, important quality differences may remain in the products themselves or in the delivery schedules (e.g. delivering earlier or later can have large price implications for suppliers). A crucial aspect of quality differences, especially impacting on equipment, is that the level of the hierarchical product classification used in the analysis may average over a range of important quality differences. Furthermore, missing information on payments and contract implementation raises questions about the validity of findings given that payments may well deviate substantially from contracted values. On a methodological level, even though our models have performed exceptionally well in predicting prices on unseen data, we cannot claim to have identified causal relationships. Without more careful identification of causal effects, by for example using quasi experimental methods, the methodology can only capture correlations and has to rely on theory and case study methodology for further support. From a policy perspective, while savings scenarios and predictors are actionable and directly relate to policy decisions made by procurement agencies, ministries of finance or individual 33 buyers, achieving and successfully implementing savings strategies remain challenging on their own. Overcoming resistance of front-line staff or upskilling key procurement officials may be more challenging than it seems at first. 34 Bibliography Bandiera, O., Best, M. C., Khan, A., & Prat, A. (2019). Incentives and the Allocation of Authority in Organizations: A Field Experiment with Bureaucrats. In American Economic Association Annual Meeting. Atlanta, Georgia, USA: AEA. Bandiera, O., Prat, A., & Valletti, T. (2009). Active and Passive Waste in Government Spending: Evidence from a Policy Experiment. American Economic Review, 99(4), 1278–1308. https://doi.org/10.1257/aer.99.4.1278 Berndt, Ernst, R. (2002). Pharmaceuticals in U.S. Health Care: Determinants of Quantity and Price. Journal of Economic Perspectives, 16 (4): 45-66., DOI: 10.1257/089533002320950975 Best, M. C., Hjort, J., & Szakonyi, D. (2017). Individuals and Organizations as Sources of State Effectiveness (NBER Working Paper Series No. No. 23350). National Bureau of Economic Research Working Paper Series. https://doi.org/10.3386/w23350 Collier, P., Kirchberger, M., & Söderbrom, M. (2015). The cost of road infrastructure in low and middle income countries. Coviello, Decio, and Mario Mariniello (2014). Publicity requirements in public procurement: Evidence from a regression discontinuity design. Journal of Public Economics 109, 2014: 76-100. Chernew, M.E. and May, D. (2011). Health Care Cost Growth. The Oxford Handbook of Health Economics. DOI: 10.1093/oxfordhb/9780199238828.013.0014 Danzon, P.M., Chao, L.W. (2000), Cross-national price differences for pharmaceuticals: how large, and why?, Journal of Health Economics, Volume 19, Issue 2, 2000, Pages 159-195, ISSN 0167-6296, https://doi.org/10.1016/S0167-6296(99)00039-9. De Oliveira, A.; Fabregas, A.; & Fazekas, M. (2019): Strategic Sourcing 2.0: Improving Fiscal Efficiency Using Big Data. Conference paper: ”Public Procurement: Global Revolution IX” at the University of Nottingham, June 2019. Decarolis, F., Giuffrida, L. M., Iossa, E., Mollisi, V., Spagnolo, G. (2020). Bureaucratic Competence and Procurement Outcomes, The Journal of Law, Economics, and Organization, Volume 36, Issue 3, November 2020, Pages 537–597, https://doi.org/10.1093/jleo/ewaa004 Fazekas, M., & Czibik, Á. (2021). Measuring regional quality of government: the public spending quality index based on government contracting data, Regional Studies, DOI: 10.1080/00343404.2021.1902975 Fazekas, M., & Blum, J. R. (2021). Improving public procurement outcomes: review of tools and the state of evidence base. Washington, DC. Fazekas, M., & Kocsis, G. (2017). Uncovering High-Level Corruption: Cross-National Corruption Proxies Using Government Contracting Data. British Journal of Political Science. 35 Fazekas, M., & Tóth, B. (2016). Assessing the potential for detecting collusion in Swedish public procurement. Stockholm. Fazekas, M., & Tóth, B. (2018). The extent and cost of corruption in transport infrastructure. New evidence from Europe. Transportation Research Part A: Policy and Practice, 113. https://doi.org/10.1016/j.tra.2018.03.021 Grennan, M. (2013). Price Discrimination and Bargaining: Empirical Evidence from Medical Devices. American Economic Review, 103 (1): 145-77. DOI: 10.1257/aer.103.1.145 James, G., Witten, D., Hastie, T., and Tibshirani, R. (2015) An Introduction to Statistical Learning: With Applications in R. 6th edition, Springer. Kohler, J.C., Dimancesco, D. (2020) The risk of corruption in public pharmaceutical procurement: how anti-corruption, transparency and accountability measures may reduce this risk, Global Health Action, 13:sup1, 1694745, DOI: 10.1080/16549716.2019.1694745 Kohler, J.C., Mitsakakis, N., Saadat, F. et al. (2015). Does Pharmaceutical Pricing Transparency Matter? Examining Brazil’s Public Procurement System. Global Health 11, 34, 2014. https://doi.org/10.1186/s12992-015-0118-8 Lehne, J., Shapiro, J. N., & Eynde, O. Vanden. (2018). Building connections: Political corruption and road construction in India. Journal of Development Economics, 131, 62–78. Lewis-Faupel, S., Neggers, Y., Olken, B. A., & Pande, R. (2016). Can Electronic Procurement Improve Infrastructure Provision? Evidence from Public Works in India and Indonesia. American Economic Journal: Economic Policy, 8(3), 258–283. Morton, F. A. & Kyle, M. (2011) Markets for Pharmaceutical Products. In Mark V. Pauly, Thomas G. Mcguire, Pedro P. Barros (2011) Handbook of Health Economics. Vol. 2., chapter 12, pages 763-823, Elsevier Petersen, O.H., Jensen, M.D. & Bhatti, Y. (2020): The effect of procurement centralization on government purchasing prices: evidence from a field experiment, International Public Management Journal, DOI: 10.1080/10967494.2020.1787278 Pricewaterhouse Coopers. (2013). Identifying and reducing corruption in public procurement in the EU. Brussels: PricewaterhouseCoopers and Ecorys. Simona Baldi & Davide Vannoni (2017) The impact of centralization on pharmaceutical procurement prices: the role of institutional quality and corruption, Regional Studies, 51:3, 426-438, DOI: 10.1080/00343404.2015.1101517 World Bank (2020) The Cost Of Staying Healthy. Semiannual Report of the Latin America and the Caribbean Region. World Bank, Washington, DC. Wouters, O. J. and Kanavos, P. G. (2017) A comparison of generic drug prices in seven European countries: a methodological analysis. BMC Health Services Research, 17 (242). pp. 1- 7. ISSN 1472- 6963 36 Yakovlev, A., Bashina, A., & Demidova, O. (2014). The effectiveness of simple homogeneous commodity procurement under rigid governmental regulation: the case of granulated sugar procurement in Russia (HSE Working papers No. WP BRP 13/PA/2014). Moscow: National Research University - Higher School of Economics. https://doi.org/DOI: Yakovlev, A., Vyglovsky, O., Demidova, O., & Bashlyk, A. (2015). Incentives for Repeated Contracts In Public Sector: Empirical Study Of Gasoline Procurement In Russia (HSE Working papers No. WP BRP 24/PA/2015). Moscow: National Research University - Higher School of Economics. 37 ANNEXES ANNEX A-Product code matching overview Table A1. High-level list of health care related product codes by country Classificati Country on level Classification description Categories excluded instrumentos de optica y aparatos y equipo fotograficos, y sus partes, piezas y accesorios aparatos medicos, instrumentos opticos y de precision, relojes y sus partes y Ecuador L2 relojes piezas Ecuador L3 productos farmaceuticos Uruguay L3 productos quimicos medicinales y farmaceuticos otros equipos tecnicos y Uruguay L2 equipos medicos, sanitarios, odontologicos y cientificos cientificos prod. quim. y conexos-medicamentos y antisepticos uso Uruguay L2 humano equipo de laboratorio, medicion, observacion y Peru L1 comprobacion Peru L1 equipos, accesorios y suministros medicos Peru L1 medicamentos y productos farmaceuticos Panama L1 equipo de laboratorio, medida, observacion y comprobacion Panama L1 medicamentos y productos farmaceuticos Panama L1 equipo, accesorios y suministros medicos Paraguay L1 equipo de laboratorio medida, observacion y comprobacion Paraguay L1 medicamentos y productos farmaceuticos Paraguay L1 equipos accesorios y suministros medicos Costa Rica L1 equipo de laboratorio medida, observacion y comprobacion Costa Rica L1 medicamentos y productos farmaceuticos Costa Rica L1 equipos accesorios y suministros medicos Brazil equipamentos e artigos para uso medico, dentario e (federal) L1 veterina-rio 38 Brazil (federal) L1 instrumentos de medicao Brazil (federal) L1 instrumentos e equipamentos de laboratorio Brazil (federal) L1 substancias e produtos quimicos Amazonas L1 aparelhos de medicao e orientacao aparelhos, equipamentos, utensilios medico-odontologicos, Amazonas L1 laboratorial e hospitalar Amazonas L1 materiais e equipamentos para medicina alternativa Amazonas L1 materiais e medicamentos para uso veterinario Amazonas L1 servicos medico-hospitalares, odontologicos e laboratoriais aparelhos, equipamentos, utensilios medico-odontologicos, Amazonas L1 laboratorial e hospitalar Amazonas L1 servicos medico-hospitalares, odontologicos e laboratoriais Amazonas L1 material laboratorial Amazonas L1 material quimico Amazonas L1 material farmacologico Santa Catarina L1 equipamentos,instrumentais e materiais de uso medico Santa Catarina L1 medicamentos Santa Catarina L1 veterinaria Santa Catarina L1 equipamentos, instrumentos e materiais odontologicos Santa Catarina L1 laboratorio, equipamentos e instrumentacao Santa Catarina L1 produtos e componentes quimicos e biologicos 39 ANNEX B. Interacted OLS Model Controls Controls + Direct Controls + Direct + Indirect policy policy Log item unit price Log item unit price Log item unit price (USD) (USD) (USD) Bidder number by country, Baseline: bidder nr=1 Country1 bidder nr=2 -0.789 *** -0.333 ** (0.000) (0.003) Country1 bidder nr=3 0.213* 0.0391 (0.022) (0.693) Country1 bidder nr=4 -0.0835 0.413** (0.541) (0.003) Country1 bidder nr=5-8 -0.109 0.486*** (0.303) (0.000) Country1 bidder nr=9-19 0.201 0.859*** (0.168) (0.000) Country2 bidder nr=2 0.134 0.133 (0.092) (0.090) Country2 bidder nr=3 0.166* 0.152 (0.036) (0.050) Country2 bidder nr=4 0.286*** 0.279*** (0.000) (0.001) Country2 bidder nr=5-8 0.485*** 0.479*** (0.000) (0.000) Country2 bidder nr=9-45 0.781*** 0.770*** (0.000) (0.000) Country3 bidder nr=2 0.0649 -0.0670 (0.170) (0.157) Country3 bidder nr=3 -0.166 * -0.283 *** (0.016) (0.000) Country3 bidder nr=4 -0.461 *** -0.578 *** (0.000) (0.000) Country3 bidder nr=5-8 -0.441 *** -0.553 *** (0.000) (0.000) Country3 bidder nr=9-54 -1.118 *** -1.184 *** (0.000) (0.000) Country4 bidder nr=2 -0.273 -0.0456 (0.408) (0.906) Country4 bidder nr=3 -0.666 -0.501 (0.055) (0.217) Country4 bidder nr=4 -0.685 -0.733 (0.149) (0.158) 40 Country4 bidder nr=5-8 -1.107 *** -1.158 *** (0.000) (0.001) Country4 bidder nr=9-13 -1.196 -1.025 (0.074) (0.141) Country5 bidder nr=2 -0.439 * -0.576 ** (0.018) (0.002) Country5 bidder nr=3 -0.553 * -0.787 ** (0.023) (0.002) Country5 bidder nr=4 0.404 0.177 (0.354) (0.687) Country5 bidder nr=5-8 0.273 0.549 (0.586) (0.279) Country6 bidder nr=2 -0.427 *** -0.506 *** (0.000) (0.000) Country6 bidder nr=3 -0.290 *** -0.404 *** (0.001) (0.000) Country6 bidder nr=4 -0.517 *** -0.683 *** (0.000) (0.000) Country6 bidder nr=5-8 -0.875 *** -0.919 *** (0.000) (0.000) Country6 bidder nr=9-75 -0.464 *** -0.807 *** (0.000) (0.000) Country7 bidder nr=2 0.0230 0.0198 (0.779) (0.807) Country7 bidder nr=3 -0.129 -0.154 (0.230) (0.146) Country7 bidder nr=4 -0.277 * -0.245 (0.030) (0.053) Country7 bidder nr=5-8 -0.456 *** -0.459 *** (0.000) (0.000) Country7 bidder nr=9-15 -0.246 -0.274 (0.227) (0.178) Country8 bidder nr=2 -0.234 0.233 (0.327) (0.357) Country8 bidder nr=3 0.160 0.204 (0.598) (0.504) Country8 bidder nr=4 -0.0522 0.354 (0.859) (0.246) Country8 bidder nr=5-8 -0.306 0.223 (0.150) (0.330) Country8 bidder nr=9-47 0.176 0.465 * (0.380) (0.029) Submission period by country, Baseline 1-7 0 days (.) Country1 subm. p.=8-12 days 0.959 *** (0.000) Country1 subm. p.=13-182 days 1.112 *** (0.000) Country1 subm. p.=missing -0.0223 41 (0.906) Country2 subm.p.=8-12 days 0.0943 (0.317) Country2 subm.p.=13-166 days 0.255 * (0.010) Country2 subm.p.=missing -0.101 (0.769) Country3 subm.p.=8-12 days -0.0379 (0.414) Country3 subm.p.=13-183 days 0.0552 (0.297) Country3 subm.p.=missing -1.235 *** (0.000) Country4 subm.p.= 8-12 days 0.323 * (0.017) Country4 subm.p.= 13-45 days 0.313 (0.343) Country4 subm.p.= missing 0.606 (0.096) Country5 subm.p =8-12 days 0.257 (0.435) Country5 subm.p =13-140 days 2.351 *** (0.000) Country5 subm.p =missing 1.435 *** (0.000) Country6 subm.p.= 8-12 days 0.301 * (0.027) Country6 subm.p =13-151 days 0.0972 (0.451) Country6 subm.p = missing 0.514 ** (0.005) Country7 subm.p=9-12 days -0.0686 (0.889) Country7 subm.p =13-132 days 0.320 (0.479) Country7 subm.p = missing 0 (.) Country8 subm.p =8-12 days 0.390 (0.379) Country8 subm.p =13-52 days 0.828 (0.133) Country8 subm.p =missing 0.153 (0.346) Procedure type, Baseline: fully competitive 0 (.) Country1 restricted competition 0 (.) Country1 non-competitive 0.258 (0.166) 42 Country1 other 2.392 *** (0.000) Country1 missing 0 (.) Country2 missing 0.0207 (0.951) Country3 restricted competition -0.640 *** (0.000) Country3 non-competitive -0.901 *** (0.000) Country3 other 0 (.) Country3 missing 0 (.) Country4 restricted competition -2.402 *** (0.000) Country4 non-competitive -1.890 *** (0.000) Country4 other -3.200 *** (0.000) Country4 missing 0 (.) Country5 restricted competition 1.192 *** (0.000) Country5 non-competitive 0.811 ** (0.002) Country6 restricted competition -1.076 *** (0.000) Country6 non-competitive -0.498 (0.122) Country8 restricted competition 0 (.) Country8 non-competitive 0.141 (0.821) Framework agreement, Baseline: No 0 (.) Country1 Framew.agreem.=Yes -0.434 *** (0.001) Country3 Framew.agreem.=Yes -2.714 ** (0.003) Country7 Framew.agreem.==1 0.542 * (0.011) Month of purchase, Baseline: December Country1 January -0.0900 (0.582) Country1 February -0.583 *** (0.000) Country1 March -0.263 43 (0.109) Country1 April -0.333 * (0.031) Country1 May -0.366 * (0.017) Country1 June -0.383 * (0.011) Country1 July -0.254 (0.065) Country1 August -0.125 (0.382) Country1 September -0.255 (0.088) Country1 October -0.0689 (0.651) Country1 November -0.353 * (0.018) Country1 missing 0 (.) Country2 January 0.0556 (0.654) Country2 February -0.00859 (0.927) Country2 March 0.0145 (0.879) Country2 April -0.0507 (0.594) Country2 May -0.240 * (0.012) Country2 June -0.0368 (0.670) Country2 July -0.0509 (0.567) Country2 August 0.196* (0.035) Country2 September 0.0720 (0.413) Country2 October 0.0962 (0.245) Country2 November -0.0348 (0.707) Country2 missing 0 Country4 January 0.188 (0.363) Country4 January 0.110 (0.584) Country4 February -0.0936 (0.606) Country4 March 0.102 44 (0.571) Country4 April -0.124 (0.494) Country4 May 0.0601 (0.742) Country4 June -0.00547 (0.976) Country4 July 0.199 (0.258) Country4 August -0.0667 (0.707) Country4 September 0.0543 (0.760) Country4 October 0.00216 (0.991) Country4 November 0 (.) Country5 January 0.155 (0.717) Country5 February 1.138* (0.019) Country5 March -0.532 (0.214) Country5 April -0.0883 (0.842) Country5 May -0.850 * (0.039) Country5 June -0.871 * (0.035) Country5 July -1.715 *** (0.000) Country5 August -1.134 ** (0.002) Country5 September -0.535 (0.159) Country5 October 0.0481 (0.904) Country5 November -0.753 * (0.023) Country5 missing 0 (.) Country6 January 1.603*** (0.000) Country6 February 0.538*** (0.000) Country6 March -1.184 *** (0.000) Country6 April 0.405*** 45 (0.001) Country6 May 0.0458 (0.661) Country6 June 0.0982 (0.290) Country6 July -0.0604 (0.481) Country6 August -0.280 ** (0.005) Country6 September -0.170 (0.056) Country6 October -0.144 (0.111) Country6 November -0.237 ** (0.009) Country6 missing 0 (.) Country7 January 0.678* (0.014) Country7 February -0.232 (0.306) Country7 March 0.116 (0.499) Country7 April 0.447* (0.010) Country7 May 0.234 (0.101) Country7 June -0.0568 (0.706) Country7 July 0.222 (0.119) Country7 August 0.569*** (0.000) Country7 September 0.782*** (0.000) Country7 October 0.390** (0.006) Country7 November 0.265 (0.080) Country7 missing 0 (.) Country8 January -0.125 (0.780) Country8 February -0.253 (0.546) Country8 March -0.00442 (0.991) Country8 April 0.109 (0.784) 46 Country8 May -0.322 (0.393) Country8 June -0.578 (0.129) Country8 July -0.196 (0.603) Country8 August 0.392 (0.298) Country8 September 0.369 (0.354) Country8 October -0.0300 (0.943) Country8 November 0.0924 (0.840) Country8 missing 0 (.) Supplier specialization Y Y Y Buyer-supplier location Y Y Y Company size Y Y Y Market concentration Y Y Y Buyer spending concentration Y Y Y Bidder number Y Y Y Avg. failed tenders % Y Y Y Avg. decision period Y Y Y Advertisement period length Y Y Y Procedure type Y Y Y Framework agreement Y Y Y Month of spending Y Y Y Product bundling Y Y Y Quantity of purchased items Y Y Y Buyer type Y Y Y Buyer state Y Y Y Contract award year Y Y Y Market ID Y Y Y Constant 6.883*** 5.590*** 3.885 *** (0.000) (0.000) (0.000) Observations 211963 34250 34250 R2 0.879 0.883 0.887 *Note: categories marked in yellow are closest matches of the reference category (Ecuador) Categories marked in orange are higher level categories compared to reference category (Ecuador) 47 ANNEX C. Random Forest details Figure C1. Number of trees and out-of-bag Mean Squared Error Table C1. Mean Squared Error and R2 as a function on number of variables sampled (m), Ntree=150, training data set m MSE % variance explained 3 1.60 82.17 4 1.52 82.99 5 1.48 83.45 6 1.46 83.67 48 7 1.46 83.75 8 1.46 83.75 9 1.47 83.65 49