WPS5528 Policy Research Working Paper 5528 Do Procurement Rules Impact Infrastructure Investment Efficiency? An Empirical Analysis of Inversão das Fases in São Paulo State Luis Blancas Laura Chioda Tito Cordella Alexandre Oliveira Felix Várdy The World Bank Procurement Unit of the Operations Service Department Brazil Country Management Unit & Latin American and the Caribbean Region Office of the Chief Economist January 2011 Policy Research Working Paper 5528 Abstract As a means to reduce delays in public works dimensions: (i) procurement process duration; (ii) implementation, a number of Brazilian states have likelihood of complaint resolution litigation; and (iii) recently reformed their procurement rules allowing prices paid to contractors. The analysis finds that the contractor price proposals to be assessed before the reform is associated with a 24 day reduction in the technical evaluation of submitted bids is undertaken duration of procurement processes for large projects (in a procedure known as inversão das fases). In order and a 7 percentage point drop in the likelihood of court to evaluate the effects of such reform, this paper adopts challenges irrespective of project size. Although both a difference-in-differences methodology to compare effects are economically important, only the latter is the procurement performance of São Paulo state (a statistical significant. Finally, the paper finds no evidence reformer state) and Minas Gerais' (a non-reformer state) of an effect of the procurement reform on prices paid. largest water and sewage utility along three efficiency This paper is a product of the Brazil Country Management Unit, the Procurement Unit of the Operations Service Department, and the Office of the Chief Economist, Latin American and the Caribbean Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at lblancas@worldbank.org, lchioda@worldbank.org, tcordella@worldbank.org, and aoliveira@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Do Procurement Rules Impact Infrastructure Investment Efficiency? An Empirical Analysis of Inversão das Fases in São Paulo State By: Luis Blancas1, Laura Chioda1, Tito Cordella1, Alexandre Oliveira1, Felix Várdy2,3 T 1 The World Bank, 1818 H Street N.W., Washington DC 20433. 2 University of California Berkeley Haas School of Business, 2220 Piedmont Avenue, Berkeley, CA 94720. 3 The authors gratefully acknowledge participants at seminars at SABESP, COPASA, the Brazilian Ministry of Planning, and the state administration of Minas Gerais for helpful comments and discussion. They also thank those institutions for their generosity in sharing data and answering to a number of queries The are deeply indebted to Rafael Arantes, Estelle Cantillon, Carlos Carvalho, Aline Dieguez, Loreni Foresti, Francisco Gaetani, Ana Hirle, Carlos Henrique, Atsushi Imi, Álvaro Mendes, Alessandro Nirino, Loçanny Silva, Karen Silverwood-Cope and Maurício Soares for useful comments and constructive criticism; and to Jivago Ximenes for his outstanding research assistance. 1. Introduction Timely investment in infrastructure remains a key enabler of Brazilian economic growth (Ferreira and Gonçalves, 2005). In practice, however, the execution of public works--which are the backbone of infrastructure investment--is a protracted and delay- prone endeavor in the country (Frischtak, 2008). Among the various stages involved in the public works execution chain, one of the most heavily regulated (through federal- and state-level laws) is the procurement of construction services. As a result, procurement tends to be seen as a unique area of opportunity to drive efficiencies in the infrastructure investment cycle through regulation reform. To what extent are procurement rules responsible for delays in the implementation of infrastructure projects in Brazil? Is it possible to improve the timeliness of Brazilian public investments by changing procurement rules? What are the effects of different procurement rules on the prices paid to contractors for the delivery of public works? In order to shed some light on these issues, in this paper we assess the effects of a prominent reform to the Brazilian open competitive bidding procurement method for public works. Commonly referred to as inversão das fases, or bid assessment stage inversion, the reform has recently been implemented in four Brazilian states (and is under consideration by Congress at the federal level). The main innovation brought about by inversão das fases vis-à-vis the "conventional" Brazilian public works procurement method is that it allows bid price proposals to be opened and taken into consideration before conducting the bids' technical evaluation. Such additional information disclosure, in principle, should reduce the amount of time spent on the technical evaluation of submitted bids, as well as the time spent in legal delays arising from the handling of complaints associated with the procurement process. Furthermore, the combination of price and technical information as inputs to the bid evaluation process may influence the prices paid by government agencies for the construction of works. The main contribution of this paper is the evaluation of a public procurement reform utilizing rigorous empirical analysis. Specifically, we study the impact of inversão das fases along three dimensions of procurement efficiency: (i) procurement process duration; (ii) likelihood of procurement process complaint resolution litigation; and (iii) prices paid to contractors. To do so, we utilize a new dataset describing public works procurement processes carried out by the largest (and majority state-owned) water and sewage utilities in two Brazilian states--SABESP, of the state of São Paulo, and COPASA, of Minas Gerais-- for the years 2007 and 2009. Since 2008 marks the year in which the state of São Paulo reformed the procurement of public works through inversão das fases, we are able to evaluate the reform's impact at SABESP utilizing a difference-in-differences methodology relative to COPASA in Minas Gerais, a state where no such reform took place. Page | 2 Our empirical analysis shows that, while the adoption of inversão das fases led to a 24 day reduction in the average duration of the procurement process for large works, such effect is not statistically significant. The reform, however, does appear to have reduced the likelihood of court challenges, by a statistically-significant 7 percentage points. Finally, the reform does not seem to be associated with any significant change in the cost of public works to taxpayers, as measured by prices paid to winning contractors. The paper is organized as follows. The next section describes the Brazilian procurement process for public works. In particular, it discusses the process's virtues and shortcomings, and the rationale by which some states have decided to reform it through the adoption of inversão das fases. Section 3 briefly discusses the theoretical literature on procurement methods, and provides some testable implications on the possible effects of inversão das fases on our three efficiency dimensions of interest: procurement process duration, incidence of process-triggered legal complaints, and prices paid to contractors. Section 4 presents the methodology used to evaluate the procurement reform in São Paulo state and further illustrates the data utilized. Section 5 presents the paper's empirical results, while section 6 provides some tentative conclusions and discusses possible extensions to the analysis. 2. Government procurement in Brazil: Why it matters In January 2007, the Brazilian federal government launched an economic stimulus package--referred to as Programa de Aceleração do Crescimento (PAC), or Growth Acceleration Program--that would spend R$504 billion (about $270 billion) over 4 years, primarily on expanding or updating the country's infrastructure stock (about 65% of planned PAC funds were explicitly allocated to infrastructure projects, with the bulk of the balance going to housing finance). Launched well in advance of the onset of the global financial and economic crisis of 2008-2009, the PAC's primary objective was to promote faster long-term growth on an equitable and fiscally responsible basis. Due to PAC's magnitude and the political prominence it has mustered, public sector efficiency in implementing PAC projects has been closely observed by a broad list of stakeholders, from political parties to oversight agencies, government contractors, the media, and, of course, the taxpayer. There are two performance indicators that these interest groups track with particular care: (i) the program's infrastructure project completion rate; and (ii) the amount spent in completed infrastructure projects as a percentage of the program's overall infrastructure budget. Yet, according to PAC's own progress reports, implementing agencies at various levels of the Brazilian public sector have experienced delays--as measured by both of these indicators. Specifically, as of April 2010 (just 8 months prior to PAC's phase-out), a full 43% of planned infrastructure projects had not been completed. Moreover, the completed projects at that point in time Page | 3 represented R$144 billion, or merely 34% of the total PAC budget for infrastructure4 (Programa de Aceleração do Crescimento, 2010). Most major public works funded by PAC (primarily in transport and energy infrastructure) are procured and contracted under Brazilian law through open competitive bidding. Since conducting bidding processes of this kind in Brazil is procedurally time consuming, it is hardly surprising that over the past three years public procurement has received a great deal of attention by stakeholders preoccupied with the timely and efficient use of stimulus money. To some of them--including the federal government-- the bottlenecks that have plagued many PAC-originated bidding events are a key reason why disbursements and project deliveries have experienced delays. The significance of public procurement efficiency in the country has only been amplified by the March 2010 unveiling of PAC 2, a second and bigger public investment program, which will spend R$959 billion (about $512 billion) over 2011-2014. 2.1 The Brazilian process for the procurement of public works The Brazilian government procurement system is governed by a two-pronged regulatory framework. The first prong is that of law 10.520 of 2002 (10.520/02), which regulates the procurement of off-the-shelf goods and non-consulting services (the latter also exclude construction services). Under this law, goods and non-consulting services are to be procured through the reverse auction method, commonly known to Brazilians as Pregão.5 The second prong in the regulatory framework is that of law 8.666 of 1993 (8.666/93) which, since the 2002 adoption of reverse auctions, has primarily been used for the procurement of public works (such as those funded by PAC) and consulting services. In the particular case of public works procurement (as opposed to consulting services), law 8.666/93 stipulates that project contracts shall be awarded on the basis of the lowest offered price among those proposals previously deemed compliant with the bidding documents. In order to determine whether a bid complies with the bidding documents (referred to as a "responsive bid"), and eventually determine how price- competitive a responsive bid is, public works procurement, as dictated by law, is to be carried out via a sequential "two-envelope" bidding process. The Brazilian two-envelope procedure for the procurement of public works takes place as follows. Having carefully reviewed the bidding documents, which describe in 4 PAC's total budget was expanded since its inception to R$656.5 billion (as of April 2010), of which R$427.8 billion was allocated to infrastructure investments in transport/logistics, energy, and social/urban development, with the rest allocated to housing and domestic sewage. 5 A reverse auction is a procurement mechanism that utilizes the lowest bid price as the critical award criterion. During the auction, bidders compete for the right to sell a given item to one or more contracting agencies by offering decreasing prices through successive, dynamic bidding, and for a pre-specified period of bidding time (say, 1 hour). When the bidding time expires, the winning bidder is the one that offered the lowest price--provided that, upon subsequent inspection, the item(s) and terms offered by the bidder are substantially responsive to the requirements set forth in the bidding documents. In Brazil, the vast majority of reverse auction events are carried out online (for example, in 2009, 97% of Federal Government reverse auctions took place over the internet). Page | 4 detail the project in question (including, for example, engineering designs), interested contractors submit their bids in two separate envelopes. The first envelope contains the bidder's legal, fiscal, technical, and financial qualification documents. Such qualification documents are meant to certify that the contractor is (i) legally eligible to win the contract, and (ii) able (technically and financially) to successfully execute it (per the detailed engineering design and estimated construction costs provided in the bidding documents). The second envelope solely contains the contractor's price proposal. After receiving paired sealed envelopes from all bidders, a technical committee appointed by the implementing agency (say, a federal- or state-government ministry) evaluates the bids in two stages. In the first stage, only the envelopes containing the qualification documents are opened and their contents assessed in order to determine which bidders are qualified to execute the contract (in strict accordance with bidding document criteria). This assessment is conducted for all bids received, regardless of what the corresponding price envelope might contain. Since reviewing bidder qualifications (particularly the technical component) can be a highly specialized (and therefore time consuming) task, the need to review all proposals may result in time bottlenecks. Bidders deemed not in compliance with the bidding documents are disqualified at the first stage and their price envelopes returned unopened. In effect, the first stage acts like a filter through which only qualified bidders will advance to the second stage, where award decisions are made. It is in this sense that the two-envelope mechanism is sometimes referred to as a pre-qualification procurement method. During the second stage, the price envelopes of qualified bidders are opened, and the qualified bidder that submitted the lowest price proposal is awarded the contract. 2.2 Distinction between evaluating project quality and evaluating bidders' qualifications In order to correctly assess the efficiency of the Brazilian two-stage procurement procedure for public works as defined by law 8.666/93, it is important to emphasize that its objective is to award a given contract to the qualified bidder that offers the lowest price. In this context, being a "qualified" bidder means, as mentioned earlier, being "able to execute" a given (i.e., pre-defined, per detailed engineering designs) project. Since a bidder can only be deemed either able or unable to execute a project, qualification decisions are strictly made on a pass/fail basis. A project's intrinsic characteristics--such as its physical layout, the materials with which it is built, the layers of reinforcement that it calls for, the project's location-- encompass what is defined as the project's "quality" dimension. These are critical, since they define how the project will ultimately fulfill the service delivery needs that led to its creation in the first place. However, such characteristics are not the object of assessment in the two-envelope procedure for the construction of public works that this paper is concerned with. Such characteristics are defined by a project's engineering design and Page | 5 estimated costs, which are an input to--not a result of--the two-envelope competitive bidding process for the procurement of public works construction.6 2.3 Complaint resolution mechanisms in the procurement of public works According to Brazil's Constitution, all government documents must be made publicly available, including those used for government procurement. Therefore, any bidder--in fact, any citizen--is allowed to inspect each and every bid received by the contracting agency during the bid opening ceremony. Equipped with that knowledge, bidders in disagreement with first-stage evaluation decisions may file complaints, either to have their bids reinstated for the second stage or to request the rejection of someone else's bid. Interested bidders have three readily available channels to voice complaints: (i) an administrative channel, implemented by the agency carrying out the procurement process; (ii) the nation's Supreme Auditor (known as Tribunal de Contas da União, or TCU), which reports to the Legislative branch of power; and (iii) the country's regular courts of law. Even though a formal administrative complaint resolution channel is provided in the procurement law, the process in practice is perceived as ineffective with regard to settling complaints definitively (as it is carried out by the very agency whose decision originated the complaint). As a result, unsatisfied bidders tend to either file directly with the courts to settle complaints, or do so after utilizing the administrative complaint resolution process as a matter of course. In recent years, the number of complaints filed with TCU has risen considerably, to a point where TCU-filed complaints may now outnumber the cases referred to courts of law. A primary reason for TCU's popularity is that, according to clause 113 of law 8.666/93, any citizen can refer a case to TCU to be reviewed free of charge, whereas regular lawsuits involve potentially costly legal fees. Moreover, even after an unfavorable decision by TCU, bidders can still file a lawsuit at regular courts if they remain unsatisfied. Importantly from a time bottleneck perspective, neither TCU nor the courts of law are bound by decision-making timeframes. Whenever a complaint is filed for review with these entities, procurement processes are interrupted for unspecified periods of time (several-month long reviews are not uncommon). As a result, the large number of lawsuits that exists in practice leads to potentially long delays in the conduction of procurement events. In addition, the fact that TCU reviews complaints at no charge to the 6 There are cases in which the design and construction of works are bundled into a single procurement process, in a procedure known as "design and build." In Brazil, the design and build approach is mainly used for awarding large concession contracts (e.g., for power plants). Design and build, however, represents but a fraction of procurement transactions; most transactions follow the more traditional two- step approach of preparing designs first (whether in-house or on a contractual basis) and subsequently launching a bid solely for the construction stage. Design and build contracts are not analyzed in this paper. Page | 6 complainant has proven to be an incentive for frivolous (i.e., otherwise avoidable) complaints. 2.4 The concept of inversão das fases As noted earlier, reviewing all technical proposals in the first stage of the evaluation process--particularly for large scale, complex public works--is time- and resource- consuming. Such an evaluation procedure is especially onerous if one takes into account that the ultimate award criterion under which contractors are compared is their offered price (in other words, submitting a responsive bid is a necessary but not sufficient condition for award). It then follows that if price proposals were known upfront, only bids that are competitive on price (say, the three lowest) could initially undergo (the resource-intensive) technical evaluation, while high bids, which holding everything else constant would have little chance of winning the contract, could be safely ignored. Opening the price proposal envelopes upfront could therefore shorten the bid evaluation period (and thus make works procurement more cost-effective). But the technical evaluation is not the only time- and resource-intensive activity that takes place in the pre-qualification bidding process: complaint resolution also plays a role. When prices are known only after the qualification documents are reviewed, procurement processes are particularly prone to legal complaints during the first stage-- especially frivolous ones--since first-stage rejected bidders do not know how likely they are, if reinstated, to eventually win the award. Conversely, if price proposals were opened first, only bidders with a real chance of winning (i.e., only price-competitive bidders) would have incentives to file a (potentially costly and protracted) complaint. Were it in fact to lead to fewer complaints per procurement process, switching the order in which bid envelopes are opened could result in shorter (and cheaper) legal delays and therefore shorter (and more resource-efficient) overall procurement processes. As it turns out, the potentially costly time and administrative resource impact of using pre-qualification in public works procurement is becoming increasingly apparent to Brazilian authorities at all levels of government. To many of them, the (highly resource- efficient and internationally emulated) reverse auction mechanism for the procurement of off-the-shelf goods has over the past eight years provided tangible evidence of the benefits of opening a bid evaluation process with prices rather than with qualification documents, as is the practice under Pregão. As a result, several states--including such investment powerhouses as São Paulo-- have modified their public works procurement laws to allow for switching the bid evaluation stages, in a procedure known as inversão das fases (which literally translates as "phase inversion"). Such a method provides the legal basis for (i) opening and assessing the contents of price envelopes before evaluating qualification documents; and (ii) conducting technical reviews of only those bids that are price-competitive (typically, it is the three lowest bids that are automatically evaluated, leaving the fourth and subsequent bids to be evaluated if, and only if, the three lowest bids are substantially unresponsive to the bidding documents). Because under inversão das fases technical Page | 7 evaluations are conducted after price proposals have been compared, the mechanism can be defined as a post-qualification procurement procedure. Besides São Paulo, which passed inversão das fases legislation in 2008, the sate administrations of Bahia (2005), Sergipe (2006), and Paraná (2007) have all enacted sub-national laws allowing phase inversion in the context of public works procurement. Notwithstanding the successful passage of these reforms at the sub-national level, phase inversion in the procurement of public works remains a highly controversial issue in Brazil. Specifically, the topic has galvanized government officials on one side (arguing for faster and more efficient works procurement implementation) and large contractors on the other (claiming that phase inversion may result in awarding contracts to unqualified bidders, with potentially risky consequences). Those against phase inversion assert that less qualified contractors could deliver low-quality construction, request amendments to the contract price during project execution (having originally secured the contract by offering the lowest price), or altogether default on a contract mid-execution. This paper will try to shed light on the first of these opposing positions. Namely, is phase inversion really associated with faster (and therefore more efficient) works procurement implementation? 2.5 The role of prices paid Aside from its impact on the duration (and therefore on time-driven costs) of public works procurement processes (both from a bid evaluation and from a complaint- resolution standpoint), inversão das fases may impact procurement efficiency through a second dimension: the use of taxpayer money in the form of prices paid to contractors. All phase inversion laws that have been enacted by state legislatures have a provision known as fase saneadora, or "cleanup phase." The role of the cleanup phase is to legally allow bid evaluation committees to either seek clarification on, or altogether waive, small, non-material deviations--vis-à-vis the bidding documents--in the qualification documents of price-competitive and otherwise responsive bids. In that way, a competitive bid (say, the lowest offered price) can be "cleaned up" of non-material deviations (say, lack of page numbering, or a missing, but retrievable, financial statement) that otherwise would have either triggered an outright rejection by the evaluation committee or significantly increased the likelihood of legal (and potentially binding) complaints by competing bidders. The federal public procurement law (8.666/93) does not provide for a fase saneadora. As a result, the approach taken in practice by bid evaluators is overly legalistic, by which any deviation from the requirements set forth in the bidding document--material or otherwise--will likely result in outright bid rejection. From a prices paid perspective, the lack of a cleanup phase may lead to highly competitive (i.e., comparatively price- attractive) bids being rejected for non-material deviations--without the evaluation committee ever knowing of the competitive nature of the rejected bids (as price envelopes are returned unopened after a first-stage rejection). Page | 8 The presence of a fase saneadora in the practice of phase inversion can thus potentially lead to lower prices paid by implementing agencies (relative to those that would have been attained without switching the order of the evaluation stages) on the basis legally more flexible (due to the cleanup phase) decisions made by evaluation committees. On the other hand, the potential for lower prices paid created by these two procedural tools can in practice be partially or entirely offset by post-award contract amendments, which may be negotiated by a successful "low-ball" bidder during project execution. But even taking only the initial award price into account, the absolute magnitude of the savings in contractor fees that might result from practicing phase inversion to award a given project is likely to be small. This is due to homogeneity in the cost structure among contractors, especially those within a state, as well as the fact that contractor bids are prepared with full knowledge of the implementing agency's estimated project cost (which tends to be highly accurate). Both of these elements constrain the variability in price proposals for a given project. Still, added across thousands of procurement processes in (currently) four states-- particularly during the implementation of a large, infrastructure-oriented stimulus package--overall savings to the taxpayer associated with prices paid under phase inversion could be substantial. Nevertheless, no effort has yet been made in the literature to measure the presence and magnitude of a potential phase inversion impact on prices paid. This paper will attempt to do that, as explained in subsequent sections. 2.6 Potential implications for different bidder invitation procedures under inversão das fases The more time- and resource-consuming it is to assess the qualifications of a group of bidders vying for a particular public works contract, the bigger the impact inversão das fases is likely to have on both the overall duration of the procurement process and the number and duration of legal complaints associated with it. Procurement process time- and resource-intensiveness, in practice, tend to be closely associated with a project's price tag: large packages (say, the construction of a bridge or a water plant) are typically more resource intensive, from a procurement perspective, than smaller packages (say, a small rural road project). The primary reason for this is that larger projects carry higher complexity in assessing bidder qualifications, as well as higher financial and engineering risk. Under law 8.666/93, even though the two-envelope bid evaluation mechanism applies to the procurement of all public works, regardless of their size, a procedure's openness to competition does depend on size (specifically, on the estimated contract size, which implementing agencies must provide to bidders as an input to bid preparation). For public works with an estimated contract size of up to R$150,000 (about $90,000), public agencies can use an invitation procedure known as convite, or limited bidding, for which Page | 9 the direct invitation of at least 3 interested bidders (whether or not they are registered in the official list of government suppliers) is enough for launching the bidding process.7 For works above R$150,000 and up to R$1.5 million, public agencies must utilize an invitation procedure known as tomada de preços, or price inquiry, where the bidding process is open to an unlimited number of interested bidders, provided they are properly registered in the official list of government contractors.8 Finally, all public works with an estimated contract size above R$1.5 million must be tendered under an invitation procedure called concorrência, or fully-open bidding, where any bidder, registered or not, can participate.9 Because projects under tomada de preços (TP) are smaller in contract size relative to those under concorrência and because, as suggested above, larger-sized projects tend to be more procurement resource intensive, it is for concorrência procurement processes for which inversão das fases would likely have the bigger efficiency impact. However, no factual evidence exists to date to corroborate the existence of such a differentiated inversão das fases impact across tender invitation procedures. 3. Some theoretical considerations and testable implications The foundations of modern auction theory were laid by Vickrey (1961), in his seminal paper on Counterspeculation, Auctions, and Competitive Sealed Tenders. Other milestones in the literature are Myerson (1981) and Riley and Samuelson's (1981) papers on optimal auctions and the revenue equivalence theorem, and the paper by Milgrom and Weber (1982) on auctions with affiliated valuations. Krishna (2002) gives an excellent overview of the very extensive theoretical literature, while the book by Milgrom (2004) takes a more applied approach and considers practical issues in auction design. In standard auctions, the principal is a monopolistic seller who only cares about the price he obtains in the auction. In a procurement auction, on the other hand, the principal is a monopsonistic buyer of goods and services who cares both about price and quality. In such an environment, bids are necessarily multidimensional in nature. The existing literature on auctions with multidimensional bids is more limited. Some references are Che (1993), Zheng (2000), and Asker and Cantillon (2008). The common approach in these papers is to study bidding under various "scoring rules" which assign scores to each aspect of a multidimensional bid that are then added up to determine the winner of the auction. 7 The procedure is still considered open competitive bidding, since uninvited bidders may themselves request participation in a convite event up to 24 hours prior to the deadline for bid submission. 8 Non-registered suppliers interested in a tomada de preços event may request registration up to 3 days prior to the event's bid submission deadline. 9 Besides limits on the number of bidders invited and restrictions regarding official registration, these three invitation procedures may also differ in terms of the broadness with which they are advertised (for example, in a newspaper of local circulation versus a newspaper of national circulation, the official gazette, and/or online government procurement portals). Page | 10 Public works procurement systems in Brazil (both in the case of pre- and post- qualification) could be modeled as a scoring auction in which the auctioneer cares about both quality and prices. As appealing as it can be from a theoretical perspective, such specification, however, would not reflect well the Brazilian system in which the qualification criteria are set by law. Thus, in our "quasi formal" analysis we model the Brazilian public works procurement system as a first-price sealed bid auction with minimum quality standards. We think that this is a better characterization of the system, despite the fact that it might not be an optimal choice. Indeed, it is well-known (Che, 1993) that scoring auctions that use the auctioneer's actual preferences are better than price-only auctions with minimum quality standards. In what follows, we describe a simple set-up that can shed some light on the effect of pre- versus post-qualification. Since the arguments we develop to compare the different types of auctions are straightforward and the equivalence results follow directly from Riley and Samuelson (1981) or Milgrom and Weber (1981), we opted to proceed in an informal way. 3.1 Sketch of a model In order to "model" inversão das fases, we assume that bidders (participating in a first price sealed bid auction to supply some kind of good to the government) differ in two dimensions: (i) whether or not they are qualified to produce the good; and (ii) the cost they face in producing it. We further assume that, before bidding, each bidder gets to observe her own cost but not whether or not she is qualified; and that if the winning bidder is unqualified, the good produced (and paid for) is worthless to the government. Under pre-qualification, the agency first chooses which bidders qualify observing a signal of their type. Then, in the second stage, it is committed to awarding the project to the lowest bid among the qualified bidders. Under post-qualification, the agency first ranks the bidders by price from low to high and, initially, only assesses whether the lowest bidder qualifies. If she does, the project is awarded to her and the auction ends. Else, the agency moves to the second-lowest bid. This in turn implies that: Result 1: The expected number of evaluations that the implementing agency needs to undertake under post-qualification is smaller than under pre-qualification. Bidders can challenge in court any decision taken by the agency. If a decision as to (dis)qualify a bidder is challenged, the court gets to observe the type and, if warranted, overrules the agency. Note that, if the cost of suing is sufficiently low (which indeed seems to be the case in Brazil as appealing to the court of auditors is free), then each and every decision taken by the agency will be challenged in court. This in turn implies that under pre-qualification the agency takes a decision as to the qualification of every bid. When the costs of suing are sufficiently small, every decision Page | 11 will be challenged in court: not only bidders who have been disqualified will challenge their disqualification, but bidders who have been qualified have an incentive to challenge the qualification of every other bidder who has also been qualified, in the hope of reducing the number of competitors. Under post-qualification, instead, there is a strictly positive probability that the court decides that the lowest bidder is qualified, and if this is the case only one lawsuit takes place. We thus have that: Result 2: When the costs of suing are sufficiently low, the expected number of lawsuits under post-qualification is strictly smaller than under pre-qualification. Looking at the bidding profiles, it is important to notice that under both pre- and post- qualification a bidder wins the auction if and only if she offers the lowest bid among all qualified bids. Since, under both systems, all of the agency's qualification decisions are challenged in court and the court perfectly observes the qualifications of each bidder, the probability of winning remains unchanged, for every strategy profile. As a bidder's payoff is fully determined by the probability of winning the auction, and the profit conditional on winning, pre- and post-qualification are strategically equivalent and, hence, a bidding profile is an equilibrium under pre-qualification if and only if it is an equilibrium under post-qualification. Accordingly, Result 3: If the bidders' costs (and the likelihood of bids being accepted) under pre- and post-qualification are the same, then a bidding profile is an equilibrium under pre- qualification if, and only if, it is an equilibrium under post-qualification. Finally, consider the case in which--for instance, as a consequence of the introduction of the fase saneadora--in the post-qualification system the likelihood that a bid (of a given qualification responsiveness) is accepted is higher than in the pre- qualification system. This would in turn imply that, for any given bidding strategy, the probability of winning (conditional upon the bid being accepted) is lower and bidding is more aggressive under post-qualification than under pre-qualification. Notice that this effect would be further magnified if the probability that a bid is accepted depends (negatively) upon the price offered. We thus have that: Result 4: If in the post-qualification system the likelihood that a bid (of a given qualification responsiveness) is accepted as responsive is higher than in the pre- qualification system, then the equilibrium prices are lower under the post-qualification system than under the pre-qualification system. Notice that Propositions 1 and 2 trivially imply the following testable implication: Testable Implication 1: The procurement process should be faster under post- qualification than under pre-qualification; Moreover, Proposition 3 and 4 imply that: Page | 12 Testable Implication 2: One should not expect a significant difference in prices paid between pre- and post-qualification rules, unless under the latter system it is more likely that (competitive) bids qualify. In what follows we evaluate whether these testable implications hold true in our analysis of the effects of shifting from pre- to post-qualification in the practice of works procurement in Brazil. 4. Methodology We assess the impact of inversão das fases on three procurement outcomes: (i) average duration of a procurement process for public works; (ii) likelihood that one or more complaints are filed during a procurement process; and (iii) prices paid to contractors, as measured by the average size of awarded contracts. To do so, we adopt a difference-in-differences (DD) strategy, wherein changes in outcomes for a contracting entity located in a reforming state--São Paulo's SABESP--pre- relative to post-reform are compared to those of a contracting entity located in a non-reforming state--Minas Gerais' COPASA. In essence, COPASA serves as a control entity, such that its trend in outcomes pre- and post-reform is differenced out of the trend observed at SABESP, the treatment entity. Since São Paulo's phase inversion law was enacted in 2008, we use procurement processes conducted in 2007 as our pre-reform sample, and processes conducted in 2009 as our post-reform sample. 4.1 Description and justification of the DD approach Formally, we estimate models of the following form: Yst SPs Y 2009 t SPs Y 2009 t X st st [1] where Yst is the outcome--either process duration (measured as the time, in days, between the opening of the first envelope and the granting of the award), complaint incidence, or award size--in state s in year t; SPs is an indicator for whether the project takes places in the treatment state, São Paulo; Y2009t is an indicator for whether the project takes place in 2009 (i.e., post-reform); and Xst is a vector of project characteristics, such as an indicator for the procurement process invitation mechanism (tomada de preços or concorrência), month of the year, and project type fixed effects. All reais-denominated quantities are deflated to January 2007 reais by the Brazilian construction price index. The parameter captures any permanent differences in outcomes between São Paulo and Minas Gerais. For instance, includes any differences in the nature of public works or "productivity" levels at SABESP relative to COPASA. In turn, embodies the trend in outcomes between 2007 and 2009 that is common to both entities. In turn, --the difference-in-differences parameter of interest--measures the change in outcome at SABESP from pre- to post-reform above and beyond that observed at COPASA, net of Page | 13 changes that are correlated with project characteristics. With rich enough Xst, and assuming that systematic differences across the two states remain fixed from 2007 to 2009, ordinary least squares estimation of [1] will deliver causal estimates of the reform's impact on works procurement outcomes. 4.2 Data description As mentioned throughout the paper, we utilize works procurement data from two water and sewage utilities operating in Brazil: SABESP, a company that serves most of the São Paulo state market, and COPASA, the largest water utility in the state of Minas Gerais. SABESP and COPASA are similar water and sewage operators in many respects. Both companies are publicly traded and majority state-owned (and therefore subject to public procurement laws). Each provides water to at least 10 million people and had 2009 revenues in excess of $1 billion.10 Importantly from a works procurement perspective, both companies have announced sizable capital expenditures for the next several years. For 2009-2010 alone, the combined capital expenditures of both companies will stand at about $3 billion ($2 billion by SABESP, $1 billion by COPASA). They also operate in similar business and socioeconomic environments within their states, as São Paulo and Minas Gerais (a) have consistently ranked among the 10 Brazilian states with the highest per capita income over the past several years, and (b) register a Human Development Index score, as measured by the UN, that is markedly higher than the national average. The data gathered comprise the full universe of procurement processes for works carried out by each company in 2007 and 2009 under both the tomada de preços and concorrência invitation mechanisms (i.e., the largest projects by size). For each bidding process, we observe data on the type of work being bid on (which makes possible segregating projects by their nature or segment), the duration (in days) of the bidding process (from the opening of the first envelope to the award of contract), the presence (if any) and duration (in days) of legal complaints filed by bidders during the bidding process, the estimated project cost (in R$) provided by each company to the bidders during the bid preparation period, and the actual contract amount (in R$), which is the end result of the procurement process. Note that the period of study purposely excludes 2008, as this was the year that São Paulo state introduced sub-national legislation allowing phase inversion for the procurement of works. More specifically, data collected from SABESP are organized in two separate datasets: the first describing all bidding processes, the second containing data specifically related to those processes impacted by procurement complaints. Table 1 describes the fields contained in each SABESP dataset. For COPASA, a single dataset was collected, whose detailed contents are described in Table 2. 10 Specifically, SABESP provides water to 24 million people and generated 2009 revenues of $4.6 billion; COPASA serves a population of some 13 million, with 2009 revenues of $1.2 billion. Page | 14 Table 3 presents summary statistics on the outcomes of interest, broken down by company/state and year. In both 2007 and 2009, COPASA conducted almost twice as many procurement processes for works under TPs and concorrências as SABESP. The average duration of a procurement process at SABESP went from 85 days in 2007 to 68 days in 2009, while similar numbers at COPASA went from 37 to 32 days. In 2007, the likelihood of a complaint being filed for a particular project at SABESP (16%) was 75% higher than that observed at COPASA (9%). In 2009, however, the incidence of litigation was virtually the same between the companies, with a slightly lower likelihood of complaints at SABESP (13%) than at COPASA (15%). COPASA projects were roughly 60% the size of their SABESP counterparts in 2007, as measured by the median contract award size. The ratio falls to 41% in 2009. 5. Empirical analysis: Results 5.1 Results As a first pass, unadjusted estimates of in equation [1] can be constructed for all 3 outcomes of interest from the figures in Table 3. For the mean duration of a procurement process (in days), the unadjusted D-in-D estimate of the effect of reform is (68.27 ­ 85.15) ­ (31.95 ­ 37.19) = -11.64. According to this result, inversão das fases was associated with a 12 day reduction in the average duration of a SABESP procurement process for works. Analogous calculations suggest inversão das fases is associated with a reduction in complaint incidence of about 9 percentage points and a reduction in contract award size of about R$7.3 million. 5.1.1 Adjusted impact on procurement process duration As laid out in the sketch of the model, the theory provides one sharp prediction: that the procurement process should be faster under post-qualification (i.e., under inversão das fases) than under pre-qualification. We test this prediction by studying the number of elapsed days between the opening of the first envelope and the signing of the award. As mentioned above, an unadjusted benchmark estimate of the effect of inversão das fases (based on Table 3) on procurement process duration is that it indeed sped up the average process by about 12 days.11 Table 4 reports estimates of , , and in equation [1], as well as the estimated coefficients of several other controls (standard errors are robust to heteroskedasticity of unspecified form). The first column ([a]) of the upper panel provides estimates for a model that controls only for seasonal indicators and procurement event type (concorrência or TP). The model suggests that at baseline SABESP's projects lasted on average 42 days longer than COPASA's. It also suggests that the reform reduced event 11 We also estimated parametric duration models. While we noticed a gain in efficiency, the point estimates were sensitive to the underlying parametric assumption, highlighting the relevance of the robustness- efficiency tradeoff. Hence, we opted to report only OLS results, which are statistically conservative and robust estimates, imposing minimal assumptions on unobservables. Page | 15 durations by about 13 days, consistent with theory (although only the difference at baseline is statistically significant at conventional levels). It is possible, however, that such reduction arises from smaller-scale projects going up for bid at SABESP in 2009 relative to 2007. This possibility is explored in column [b], where we additionally control for each company's estimated costs for projects. Controlling for the company's "reservation price," which carries information about project scale and scope, indeed reduces the differences reported in column [a]. Even this (statistically insignificant) gain in process speed may be overstated if complaints on average lengthen durations and post-qualification generates fewer bidder complaints (which we test below). Column [c] therefore reports results of specifications that add an indicator of whether any litigation was pursued. These suggest that the reform increased delays by one statistically insignificant day and that, on average, litigated (i.e., complaint-delayed) events are 47 days longer (which is statistically distinguishable from zero at the 1% level). One concern regarding these estimates is that the composition of company-specific projects may have changed in either firm from 2007 to 2009, which may confound the effects of the reform. In the lower panel of Table 4, we address this concern by controlling for the general category of projects, adding indicators for project segment. However, all conclusions are qualitatively unchanged, with point estimates remaining relatively stable from the top to the bottom panel, suggesting that composition effects do not appear to confound the conclusions. In sum, there is no evidence that inversão das fases sped up the process when we look at the full universe of projects (that is, including concorrências and TPs). Such a result is consistent with our original conjecture that the phase inversion impact on process duration is not likely to be significant when smaller-sized (and therefore less complex) TP projects are included in the analysis. As will be shown later in detail, when restricting the sample to concorrência-only projects, the reform is associated with a 30% drop in the duration of the procurement process. While economically significant, the effect is not statistically significant--possibly due to limited sample size (there are 223 large scale projects in our sample). 5.1.2 Adjusted impact on the likelihood of complaints We then test the proposition that the likelihood of lawsuits or complaints under post- qualification is smaller than that under pre-qualification, as hypothesized above. We estimate probit versions of equation [1], where the outcome is an indicator for whether a complaint was filed, and report average marginal effects in Table 5. Although there is no statistically discernible difference in the incidence of complaints between the two companies after controlling for covariates and estimated project costs, the table's lower panel (which includes segment fixed effects) suggests that litigation is approximately 7 percentage points less prevalent post-reform. This figure is both statistically (at the 5% level) and economically significant, since 13% of projects in our sample are the subject of litigation. Page | 16 5.1.3 Adjusted impact on prices paid We end this section by estimating versions of equation [1] where the award size (in real terms) is the outcome. Table 6 reports estimates of , , and . The upper panel of column ([a]) indicates that, at baseline, SABESP's projects were on average R$7 million costlier (i.e., carried a larger award size by that amount) than those at COPASA. The project cost gap was reversed by the reform, which was associated with a R$7.6 million reduction in award size (both numbers are statistically significant at conventional levels). However, column [b], which controls for each company's a priori estimated project costs, illustrates that this reduction resulted from smaller scale projects put up for bid at SABESP in 2009 relative to 2007: holding constant each company's estimated project costs eliminates the statistically significant difference in award size reported in column [a].12 To confirm that the scale of projects changed at SABESP between 2007 and 2009, we estimate the D-in-D model on the estimated project costs and report estimates in column [e]. According to this specification, the SABESP-COPASA gap in project budgets at baseline (which averaged +R$8.5 million), was entirely closed in 2009, as hypothesized. In column [c], we additionally control for whether a complaint was filed over the procurement process. Doing so barely alters the estimates from column [b], though the point estimate on the dummy indicating the presence of one or more complaints implies that such projects are associated with roughly R$ 0.6 million lower estimated costs. Finally, in column [d] we explore the possibility of a structural break in the relationship between actual and estimated costs, by controlling for the interaction of the 2009 year indicator and the estimated cost. No such structural break is observed. As mentioned above, the possibility of company-specific projects to change segment composition from 2007 to 2009 may confound the effects of the reform. In the lower panel of Table 6 we therefore control for the general characteristics of projects by adding indicators for project segment. However, this hardly alters conclusions or point estimates relative to the top panel, suggesting that the composition of projects, as embodied by these indicators, did not change materially between 2007 and 2009 for both SABESP and COPASA. As a robustness check, and to explore the possibility that the zero effect at the mean does not mask heterogeneity in the reform's effect elsewhere on the conditional distribution of contract prices, we estimate quantile models analogous to equation [1] for the 5th, 10th, 50th, 90th, and 95th centiles of project costs. The idea being that effects that go undetected at the mean may still be observed if the reform either compressed or flattened the conditional distribution of the outcome. The quantile regression estimates are reported in Table 7 and show little effect at any of the conditional quantiles, even at the 10% level. 12 The addition of this control trivially increases the explanatory power of the model: the R2 rises from 19% in column [a] to 87% after controlling for estimated costs. Page | 17 5.2 Concorrência-only projects In 2007 and 2009, 29% of all projects in our sample were launched as concorrências (based on their estimated cost), but represented 92% of the almost R$3 billion in contracts awarded across both companies. As noted earlier, these projects are of special interest from a procurement efficiency perspective, due to their relatively higher procedural complexity. We thus repeat the above analyses for this subset of projects. In addition, this exercise allows us to indirectly explore the heterogeneity of the effects of the reform as a function of the economic scale of the underlying projects. Table 4' studies the effect of the reform on the procurement process duration among these larger scale projects (which take on average 20 days longer to implement across both companies). Conditional on segment fixed effects and estimated project costs, concorrência events took 64 additional days to be carried out at SABESP relative to COPASA. Looking at column [c] we find a statistically insignificant but economically large impact--a 24-day reduction--of post-qualification on the duration of procurement processes. Table 5' provides no statistically significant evidence that the reform affected the likelihood of complaints among concorrência projects, though the point estimate of the difference-in-differences marginal effect precisely matches the one observed for the full sample. Finally, Table 6' reports the results of the difference-in-differences analysis on prices paid for concorrência projects. At baseline, such public works were R$22 million costlier at SABESP than at COPASA (column [a]), but this gap was more than offset by the procurement reform (which is associated with a change of -R$26 million). However, as in the pooled sample case, these differences are entirely accounted for once we control for estimated project costs, such that the estimates of , , and are all indistinguishable from zero. These results emphasize a tradeoff between economic and statistical significance. When we restrict the analysis to concorrência projects, our estimates are often larger in magnitude; however the more limited number of observations does not grant enough power to identify the effects with generally-accepted precision levels. Our estimation methodology relies on a local identification strategy (i.e., in the neighborhood of the policy change) that enables us to estimate the parameter of interest without imposing structural assumptions beyond a minimal (semiparametric) linear specification. Given the limited amount of data, an alternative avenue would be to exploit more carefully the economic structure of the bidding problem to inform the estimating equation and improve the precision of our estimates (see for instance Athey and Haile, 2006), thereby relaxing the data requirement imposed by semiparametric methods. Page | 18 6. Conclusions According to economic theory, the inversão das fases procedure adopted by SABESP in São Paulo state should have led to a faster procurement process, due to both a procedurally simpler process and a reduction in the number of decision complaints. In addition, one should have expected that, if the introduction of the fase saneadora increased the number of competitive bids, then more competitive prices would also result from the reform. In order to test such theoretical predictions, we utilized a difference-in- differences methodology that allowed us to isolate and quantify the impact of the reform on works procurement efficiency. While our results suggest that inversão das fases might have sped up the procurement process, mainly through the reduction in the number of complaint filings, the results we find are not statistically significant. This may well reflect the limited number of observations with which we worked, and we hope that in future new data will allow us, as it becomes available, to improve on the significance of our estimates and get a cleaner sense of the effects of the reform. Looking at the reform's impact on the prices paid to winning bidders, we find no evidence of a reduction effect. However, such finding should be taken as very preliminary. The information available in our dataset allowed us to measure the reform's impact on the size of the award, as opposed to the impact on actual fees ultimately paid to contractors from project inception through delivery (some projects can be highly prone to contract amendments during implementation, which may materially impact the fees charged by contractors). Since we have not (yet) been able to collect data on amendment requests during project execution, our conclusion on prices paid should be seen as an approximation to the impact of the reform on "true" (i.e., ultimately incurred) prices. We hope that new data will in future allow us to improve on the significance of our estimates. Nevertheless, even with the data limitations faced we believe this paper provides a useful contribution to the procurement literature by deriving testable implications from economic theory and applying a statistically-sound methodology to the evaluation of a particular procurement reform. Page | 19 References Athey, Susan and Philip A. Haile, 2006, Empirical Model of Auctions, Invited lecture Ninth World Congress of the Econometric Society, London Asker, John and Estelle Cantillon, 2008, Properties of Scoring Auctions, RAND Journal of Economics, Vol. 39(1), pp 69-85. Che, Y.-K. (1993), Design competition through multidimensional auctions, RAND Journal of Economics 24: 668­680. Ferreira, P. and L. Gonçalves (2005),"Welfare and Growth Effects of Alternative Fiscal Rules for Infrastructure Investment in Brazil," Economics Working Papers (Ensaios Economicos da EPGE) 604, Graduate School of Economics, Getulio Vargas Foundation. Frischtak, C. (2008), O Investimento em Infra-estructura no Brasil: Histórico Recente e Perspectiva, Pesquisa e Planejamento Econômico, Vol. 38, No. 2. Harstad, Ronald M., Kagel, John H. and Levin, Dan, (1990), Equilibrium bid functions for auctions with an uncertain number of bidders, Economics Letters, Elsevier, vol. 33(1): 35-40. Krishna, Vijay, 2002, Auction Theory, Academic Press, San Diego, California. Milgrom, Paul R., 2004, Putting Auction Theory to Work, Cambridge University Press. Milgrom, Paul R. and Robert J. Weber (1981), A Theory of Auctions and Competitive Bidding, Econometrica Vol. 50, No. 5, pp. 1089-1122. Myerson, R. B. (1981), Optimal auction design, Mathematics of Operations Research 6: 58­73. Programa de Aceleração do Crescimento, 10° Balanço Janeiro a Abril de 2010, Brazilian Federal Government, 2010. Riley, John G. and William Samuelson (1981), Optimal Auctions, American Economic Review, 71, 381-392. Vickrey, William, 1961, Counterspeculation, Auctions, and Competitive Sealed Tenders, Journal of Finance. Page | 20 Table 1. Structure of the SABESP Dataset SABESP Dataset 1: Full Sample Field Description Procurement method CP for Concorrências, TP for Tomada de Preços indicator Bidding process Unique ID assigned to each bidding process identification number Works description Summary description of the nature of the works procured Segment Works projects are categorized by segment, based on activity type. There are 16 project segments, including water plants, water pipes, connections, and sewage plants Date of opening of the first For 2007 (prior to the reform), the first envelope contains envelope the technical, financial, legal, and fiscal qualification documents. For 2009 (post reform), the first envelope corresponds to the price proposal Contract signature date Date contract was signed Estimated cost Officially estimated project cost, as prepared by SABESP, in Brazilian currency Contract price Actual size of the contract awarded as a result of the procurement process, in Brazilian currency Page | 21 SABESP Dataset 2: Sub-sample of Processes Impacted by Complaint Resolution Delays Field Description Bidding process Unique ID assigned to each bidding process identification number Process stage in which Four options: complaint was filed Habilitação, or technical evaluation phase (where technical, financial, legal, and fiscal qualification documents are reviewed) Price evaluation phase Contract award Contract signature Complaint filing date Date complaint was received by SABESP or filed with TCU or a court of law Complaint closing date Date complaint was resolved Complaint resolution Number of days elapsed until the complaint was resolved duration Type of complaint Five possible types: Administrative complaint against evaluation of technical bids Administrative complaint against evaluation of prices Lawsuit against evaluation of technical bids Lawsuit against evaluation of prices Lawsuit against contract award Reason for filing complaint Optional field with open-ended description Page | 22 Table 2. Structure of the COPASA Dataset Field Description Bidding process Unique ID assigned to each bidding process identification number Procurement method Concorrência or Tomada de Preços Estimated cost Officially estimated project cost, as prepared by COPASA, in Brazilian currency Works description Summary description of the nature of the works procured Date of opening the first Date of opening the habilitação envelope, which contains envelope the technical, financial, legal, and fiscal qualification documents Date of opening the second Date of opening the envelope containing the bid price envelope Contract signature date Date contract was signed Contract price In Brazilian currency Presence of procurement Yes/No indicator of whether the procurement process was complaints delayed by the resolution of one or more complaints filed by bidders Page | 23 Table 3. Summary Statistics COPASA SABESP (Minas Gerais) (São Paulo) Procurement Median 31.00 56.00 Process Duration Mean 37.19 85.15 (days) St. Dev 21.53 82.41 Complaint Incidence Mean 0.09 0.16 2007 Median 296,671 498,408 Award Size Mean 990,563 10,000,000 (2007 R$) St. Dev 4,593,681 31,500,000 N 210 107 Procurement Median 21.00 55.00 Process Duration Mean 31.95 68.27 (days) St. Dev 29.85 47.13 Complaint Incidence Mean 0.15 0.13 2009 Median 398,843 969,577 Award Size Mean 2,431,077 4,163,833 (2007 R$) St. Dev 5,925,948 10,500,000 N 341 157 Notes: Excludes processes above 1,000 days in duration. Page | 24 Table 4. Estimated DinD Effects on Duration of Procurement Process (Excluding Durations > 1000 ­ N=815) Procurement Process Duration (days) [a] [b] [c] SP x Y2009 12.80 2.27 1.09 SE [8.53] [7.51] [6.62] SP 41.91 31.83 31.52 SE [7.34]*** [6.40]*** [5.81]*** Without Segment Fixed Effects Y2009 12.35 11.31 12.77 SE [2.23]*** [2.17]*** [1.97]*** Controlling for Method = Concorrência 34.36 20.43 12.51 SE [4.48]*** [4.26]*** [3.74]*** Estimated Project Cost 9.69E07 7.48E07 SE [3.14e07]*** [3.70e07]** Complaint Indicator 46.81 SE [6.89]*** RSquared 0.30 0.42 0.51 SP x Y2009 13.78 3.07 1.01 SE [8.64] [7.95] [6.96] SP 41.33 32.18 32.06 SE [8.07]*** [7.45]*** [6.66]*** With Segment Fixed Effects Y2009 11.30 10.85 13.02 SE [2.33]*** [2.27]*** [2.06]*** Controlling for Method = Concorrência 33.51 20.80 13.42 SE [4.19]*** [4.25]*** [3.71]*** Estimated Project Cost 8.74E07 6.20E07 SE [3.52e07]** [3.81e07] Complaint Indicator 47.40 SE [6.87]*** RSquared 0.35 0.44 0.54 Notes: Standard errors are robust to heteroskedasticity. All prices are in real 2007 Reais. Project segments are: Estacoes Elevatorias De Agua E/Ou Boosters; Estacoes De Tratamento De Agua; Adutoras; Redes De Distribuicao De Agua; Estacoes Elevatorias De Esgotos; Estacoes De Tratamento De Esgotos; Interceptores,Coletores E Emissario; Emissarios E Adutoras Subaquaticas; Redes Coletoras De Esgotos; R eservatorios; Estruturas; Pocos Profundos; Instalacoes E Montagens Industriais; Instalacoes Eletricas; Edificacoes; Ligacoes Prediais De Agua Em Redes; Ligacoes Prediais De Esgotos Em Red; Serviços Operacionais; Sistema De Abastecimento De Água; Solid Waste Facility. Page | 25 Table 5. Estimated DinD Effects on Likelihood of Complaints (Excluding Durations > 1000 ­ N=815) Linear Prob. Model Probit Marginal Effects [a] [a] [b] SP x Y2009 0.11 0.08 0.06 SE [0.05]** [0.03]*** [0.03]* SP 0.04 0.04 0.01 Without Segment Fixed Effects SE [0.04] [0.04] [0.04] Y2009 0.03 0.03 0.03 SE [0.03] [0.03] [0.03] Controlling for Method = Concorrência 0.22 0.23 0.19 SE [0.03]*** [0.03]*** [0.03]*** Estimated Project Cost 1.99E09 SE [9.34E10]** RSquared 0.10 SP x Y2009 0.13 0.09 0.07 SE [0.05]** [0.02]*** [0.03]** SP 0.04 0.03 0.01 With Segment Fixed Effects SE [0.04] [0.04] [0.04] Y2009 0.04 0.04 0.04 SE [0.03] [0.03]* [0.03]* Controlling for Method = Concorrência 0.21 0.22 0.17 SE [0.03]*** [0.03]*** [0.03]*** Estimated Project Cost 2.79E09 SE [1.06E09]*** RSquared 0.14 Notes: Standard errors are robust to heteroskedasticity. All prices are in real 2007 Reais. Project segm ents are: Estacoes Elevatorias De Agua E/Ou Boosters; Estacoes De Tratamento De Agua; Adutoras; Redes De Distribuicao De Agua; Estacoes Elevatorias De Esgotos; Estacoes De Tratamento De Esgotos; Interceptores,Coletores E Emissario; Emissarios E Adutoras Subaquaticas; Redes Coletoras De Esgotos; Reservatorios; Estruturas; Pocos Profundos; Instalacoes E Monta gens Industriais; Instalacoes Eletricas; Edificacoes; Ligacoes Prediais De Agua Em Redes; Ligacoes Prediais De Esgotos Em Red; Serviços Operacionais; Sistema De Abastecimento De Água; Solid Waste Facility. Page | 26 Table 6. Estimated DinD Effects on Award Size (Excluding Durations > 1000 ­ N=815) Estimated Award Size Project Cost [a] [b] [c] [d] [e] SP x Y2009 7,582,793 169,233 126,851 59,171 8,875,933 SE [2,997,175.32]** [191,957.29] [195,993.42] [231,247.56] [3,377,986.15]*** SP 6,969,528 445,275 441,369 367,726 8,489,817 SE [2,648,186.70]*** [149,306.26]*** [148,653.01]*** [191,513.07]* [2,945,457.41]*** Y2009 559,298 203,413 221,799 151,038 873,290 Without Segment Fixed Effects SE [556,526.66] [75,017.63]*** [84,594.83]*** [116,582.21] [648,563.89] Controlling for Method = Concorrência 10,453,008 204,010 303,717 92,506 11,734,920 SE [1,519,327.82]*** [357,051.87] [306,001.38] [213,895.50] [1,684,156.11]*** Estimated Project Cost 0.8700 0.8800 0.8700 SE [0.05]*** [0.05]*** [0.06]*** Complaint Indicator 589,072 SE [463,268.15] Y2009 x Budgeted Cost 0.0032 SE [0.06] RSquared 0.19 0.98 0.98 0.98 0.20 SP x Y2009 8,011,744 432 33,868 184,018 9,394,133 SE [2,721,245.07]*** [180,148.42] [197,333.44] [191,155.56] [3,189,452.87]*** SP 6,442,655 412,898 411,918 261,240 8,038,014 SE [2,452,810.50]*** [196,596.51]** [196,651.24]** [190,167.30] [2,906,629.42]*** Y2009 125,426 217,023 235,255 114,469 401,515 With Segment Fixed Effects SE [578,113.73] [83,330.16]*** [88,981.07]*** [98,700.43] [670,474.73] Controlling for Method = Concorrência 9,940,616 420,086 482,079 250,346 11,162,650 SE [1,248,077.58]*** [280,643.11] [260,705.23]* [167,661.19] [1,422,765.60]*** Estimated Project Cost 0.8500 0.8500 0.8400 SE [0.04]*** [0.04]*** [0.05]*** Complaint Indicator 398,339 SE [363,620.20] Y2009 x Estimated Project Cost 0.0500 SE [0.05] RSquared 0.34 0.98 0.98 0.98 0.32 Notes: Standard errors are robust to heteroskedasticity. All prices are in real 2007 Reais. Project segments are: Estacoes Elevatorias De Agua E/Ou Boosters; Estacoes De Tratamento De Agua; Adutoras; Redes De Distribuicao De Agua; Estacoes Elevatorias De Esgotos; Estacoes De Tratamento De Es gotos; Interceptores, Coletores E Emissario; Emissarios E Adutoras Subaquaticas; Redes Coletoras De Esgotos; Reservatorios; Estruturas; Pocos Profundos; Instalacoes E Montagens Industriais; Instalacoes Eletricas; Edificacoes; Ligacoes Prediais De Agua Em Redes; Ligacoes Prediais De Esgotos Em Red; Serviços Operacionais; Sistema De Abastecimento De Água; Solid Waste Facility. Page | 27 Table 7. Estimated DinD Effects on Quantiles of Award Size (Excluding Durations > 1000 ­ N=815) Estimated Award Size Project Cost [a] [b] [c] [d] [e] SP x Y2009 Centile 0.05 16,038 9,118 11,025 225,499 47,351 SE [33,644.8483] [123,469.3026] [97,736.4472] [183,517.3332] [40,235.6085] Centile 0.10 37,007 64,298 64,298 183,387 36,161 SE [32,707.1721] [89,196.4735] [198,255.5740] [165,620.1234] [36,397.9547] Centile 0.50 7,546 41,541 37,470 70,946 9,551 SE [602,185.7025] [42,081.0636] [422,170.9901] [39,724.3680]* [546,609.2781] Centile 0.90 677,436 163 477 2,031 536,544 SE [1.4144e+07] [2,065.9986] [24,898.7455] [5,316.4451] [1.8584e+07] Centile 0.95 50,470,000 309 878 1,616 61,911,000 SE [2.2893e+07]** [6,311.0326] [5,123.2546] [166,291.1770] [3.1528e+07]** SP Centile 0.05 94,253 61,669 63,577 22,110 87,899 SE [27,252.2957]*** [145,439.0074] [121,842.3936] [157,465.8044] [35,113.5865]** Centile 0.10 67,551 119,953 119,953 23,030 91,750 SE [25,706.8455]*** [97,253.4293] [271,897.7542] [140,993.2751] [29,530.4023]*** Centile 0.50 94,434 12,619 15,524 1,362 140,667 SE [533,217.4946] [24,292.8443] [29,345.0822] [25,897.2575] [498,921.7517] Centile 0.90 852,684 167 168 158 805,837 SE [1.3404e+07] [1,887.8885] [35,406.6883] [4,653.8237] [1.8553e+07] Centile 0.95 50,555,000 72 877 119 62,207,000 SE [2.2639e+07]** [1,468.9709] [31,711.1151] [2,286.5566] [3.1444e+07]** Y2009 Centile 0.05 3,091 357 424 26,253 7,535 SE [18,416.4402] [44,628.8521] [30,524.9694] [136,978.1999] [22,081.8965] Centile 0.10 15,038 3,302 3,302 26,315 14,256 SE [15,533.3010] [16,865.2635] [130,487.9669] [29,509.3426] [19,772.0017] Centile 0.50 11,660 23,564 23,949 26,431 14,588 SE [460,146.3061] [9,393.9253]** [22,452.7984] [32,088.9581] [392,503.3660] Centile 0.90 6,180 1,149 1,464 2,751 121,028 SE [3372960.4144] [1,260.5456] [10,555.2057] [10,394.6897] [1282742.0122] Centile 0.95 78,829 258 283 934 175,516 SE [3388879.2793] [793.5123] [6,265.3587] [5,400.2453] [388,787.7879] Controls for Method = Concorrência X X X X X Estimated Project Cost X X X Complaint Indicator X Y2009 x Estimated Project Cost X Notes: Standard errors are bootstrapped, with 300 replications. All prices are in real 2007 Reais. Page | 28 Table 4. Estimated DinD Effects on Duration of Procurement Process for Concorrência Projects (Excluding Durations > 1000 ­ N=233) Procurement Process Duration (days) [a] [b] [c] SP x Y2009 57.82 27.74 23.63 SE [25.79]** [23.04] [21.33] SP 91.31 64.17 66.78 With Segment Fixed Effects SE [25.2]*** [22.48]*** [20.35]*** Y2009 3.90 8.41 9.53 SE [10.68] [10.19] [8.7] Controlling for Estimated Project Cost 9.57E07 6.85E07 SE [3.33E07]*** [3.75E07]* Complaint Indicator 52.47 SE [10.59]*** RSquared 0.38 0.45 0.55 Notes: Standard errors are robust to heteroskedasticity. All prices are in real 2007 Reais. Project segments are: Estacoes Elevatorias De Agua E/Ou Boosters; Estacoes De Tratamento De Agua; Adutoras; Redes De Distribuicao De Agua; Estacoes Elevatorias De Esgotos; Estacoes De Tratamento De Esgotos; Interceptores,Coletores E Emissario; Emissarios E Adutoras Subaquaticas; Redes Coletoras De Esgotos; Reservatorios; Estruturas; Pocos Profundos; Instalacoes E Montagens Industriais; Instalacoes Eletricas; Edificacoes; Ligacoes Prediais De Agua Em R edes; Ligacoes Prediais De Esgotos Em Red; Serviços Operacionais; Sistema De Abastecimento De Água; Solid Waste Facility. Page | 29 Table 5. Estimated DinD Effects on Likelihood of Complaints for Concorrência Projects (Excluding Durations > 1000 ­ N=233) Complaint Indicator Probit Marginal Effects [a] [b] [a] [b] SP x Y2009 0.24 0.08 0.23 0.07 SE [0.14]* [0.15] [0.11]** [0.15] With Segment Fixed Effects SP 0.10 0.05 0.08 0.08 SE [0.13] [0.14] [0.13] [0.14] Y2009 0.05 0.02 0.06 0.03 SE [0.1] [0.1] [0.09] [0.1] Controlling for Estimated Project Cost 5.18E09 5.24E09 SE [1.71E09]*** [1.93E09]*** RSquared 0.16 0.21 Notes: Standard errors are robust to heteroskedasticity. All prices are in real 2007 Reais. Project segm ents are: Estacoes Elevatorias De Agua E/Ou Boosters; Estacoes De Tratamento De Agua; Adutoras; Redes De Distribuicao De Agua; Estacoes Elevatorias De Esgotos; Estacoes De Tratamento De Esgotos; Interceptores,Coletores E Emissario; Emissarios E Adutoras Subaquaticas; Redes Coletoras De Esgotos; R eservatorios; Estruturas; Pocos Profundos; Instalacoes E Montagens Industriais; Instalacoes Eletricas; Edificacoes; Ligacoes Prediais De Agua Em Redes; Ligacoes Prediais De Esgotos Em Red; Serviços Operacionais; Sistema De Abastecimento De Água; Solid Waste Facility. Page | 30 Table 6. Estimated DinD Effects on Award Size for Concorrência Projects (Excluding Durations > 1000 ­ N=233) Estimated Award Size Project Cost [a] [b] [c] [d] [e] SP x Y2009 26,138,300 29,248 32,901 843,007 31,432,900 SE [9,102,990]*** [947,979] [984,223] [1,004,790] [10,334,700]*** SP 22,474,000 1,090,610 1,092,930 123,336 28,369,600 SE [8,963,950]** [920,408] [901,123] [951,106] [10,327,800]*** With Segment Fixed Effects Y2009 4,981,630 1,068,920 1,069,920 274,993 4,710,530 SE [2,699,280]* [542,832]** [546,878]* [601,514] [2,863,480]* Controlling for Estimated Project Cost 0.83 0.83 0.81 SE [0.04]*** [0.04]*** [0.05]*** Complaint Indicator 46,536 SE [666,045] Y2009 x Budgeted Cost 0.09 SE [0.05] RSquared 0.51 0.99 0.99 0.99 0.47 Notes: Standard errors are robust to heteroskedasticity. All prices are in real 2007 Reais. Project segm ents are: Estacoes Elevatorias De Agua E/Ou Boosters; Estacoes De Tratamento De Agua; Adutoras; Redes De Distribuicao De Agua; Estacoes Elevatorias De Esgotos; Estacoes De Tratamento De Esgotos; Interceptores,Coletores E Emissario; Emissarios E Adutoras Subaquaticas; Redes Coletoras De Esgotos; Reservatorios; Estruturas; Pocos Profundos; Instalacoes E Montagens Industriais; Instalacoes Eletricas; Edificacoes; Ligacoes Prediais De Agua Em Redes; Ligacoes Prediais De Esgotos Em Red; Serviços Operacionais; Sistema De Abastecimento De Água; Solid Waste Facility. Page | 31