The World Bank Economic Review, 31(2), 2017, 553–569 doi: 10.1093/wber/lhw005 Advance Access Publication Date: March 22, 2016 Article Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 Providing Policy Makers with Timely Advice: The Timeliness-Rigor Trade-off Clive Bell and Lyn Squire Abstract Policy makers bemoan the lack of research findings to guide urgent decisions, whereas researchers’ professional code puts rigor first. This article argues that provisional assessments, produced early in the research cycle, can bridge the gap. Numerous case studies point to the importance of early interaction with policy makers and the delivery of brief, policy-focused papers; but preliminary analyses may be flawed and so increase the chances of a wrong decision. This article demonstrates analytically that a preliminary assessment, supported by the offer of more refined research, provides an option that is superior, on average, to the current practice of submitting a final report at the end of the research cycle. Where practical implementation is concerned, it calls for donor- funded subsidies to promote the use of provisional assessments and for a rapid, independent, professional re- view process to ensure their quality. While the research-policy exchange in developing countries is a complex, context-specific phenomenon, the proposal offered here holds out some promise of improving decisions in the public sphere under a wide range of circumstances. JEL classification: D61, H43, O20, O21, O22 If economists could manage to get themselves thought of as humble, competent people on a level with dentists, that would be splendid. —J. M. Keynes, Essays in Persuasion, 1931. I. Introduction The development community has embraced the notion of so-called evidence-based policy, which is gen- erally understood to be public policy informed by rigorously established, objective evidence (see, e.g., Pawson 2006; Nutley et al. 2007; Carden 2009). This general desideratum is certainly not new, but it was popularized lately by the Blair Government in the United Kingdom (Cabinet Office 1999a, 1999b). Official pronouncements have their value, but the link between research and policy is often tenuous at best. Clive Bell is professor of economics, emeritus, at the University of Heidelberg and Adjunct Senior Researcher at CMI, P.O. Box 6033, N-5892 Bergen, Norway; his email address is: clive.bell@urz.uni-heidelberg.de. Lyn Squire (corresponding author) is the former president of the Global Development Network, 4, Vasant Kunj Institutional Area, New Delhi-110070, India; his email address is lynsquire@yahoo.com. We are much indebted to Pierre Jacquet for valuable and extensive comments on an earlier draft. Subsequently, helpful suggestions from Fred Carden, Randy Filer, Stefan Klasen, Halvor Mehlum, Lant Pritchett, Indira Rajaraman, Alan Winters, participants in CMI’s annual development seminar, and especially two anonymous referees led to improvements in various parts of the text. We bear sole responsibility for all remaining errors of analysis and opinion. C The Author 2016. Published by Oxford University Press on behalf of the International Bank for Reconstruction and Development / THE WORLD BANK. V All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. 554 Bell and Squire The focus of this article is the interaction between researchers and policy makers in developing coun- tries. Descriptions of this process usually stress its complexity and context-specificity and point to a multi- tude of factors, other than research, that influence decisions (Court et al. 2005; Livny et al. 2006; Carden 2009). Policy makers have to worry about the interests of relevant constituents, pressure from external agencies, and personal career ambitions; they may have to contend with weak implementing agencies, cor- Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 ruption, and even violence; they may prefer to draw on their own experience and knowledge or that of their immediate advisors; and they may feel that local research is unreliable or based on data that are lim- ited, inaccurate, or unavailable when required. In sum, even where research appears to play a role, it will only be one, and usually not the most important, in a complex array of factors influencing policy. If the aim is to make research more influential, efforts to remove, or at least weaken, the impediments to a vigorous research-policy interaction merit serious consideration. The contribution of this article is to propose a way of overcoming one of these obstacles—the frequent unavailability of research when re- quired for urgent policy decisions. While research must be sufficiently rigorous that, if acted upon, there is a reasonable expectation it will improve the policy decision, it must also be timely, in the sense that it has to reach the policy maker before the decision is made. This article argues that timeliness of results has received less attention than research quality—and less than it merits. Meeting accepted professional standards has been the watchword for the research community. In con- sequence, efforts to feed advice and guidance into the policy-making process are usually undertaken only at the end of the research cycle, after the research has been completed and vetted. While there is an obvi- ous rationale for this—no one wants results based on badly flawed analysis or grossly inaccurate data reaching the policy maker’s desk—the approach runs the risk that results arrive too late to be of value. Indeed, decision makers worldwide constantly voice frustration with the slowness of the research pro- duction process. The ODI/SciDev.Net International Survey on the Science-Development Policy Inter- face, for example, reports that “A general consensus from the expert interviews was that a major chal- lenge is the narrow focus and long time-scales of scientific research compared with political priorities” (Jones et al. 2008, 20). Similarly, an ODI Briefing Paper notes that “Policymakers bemoan the inability of many researchers to make their findings accessible and digestible in time for policy decisions. Practitioners often just get on with things” (Overseas Development Institute 2004, 1). A possible remedy is to encourage the submission of preliminary results early in the research cycle in order to increase the likelihood of their reaching the policy maker before the decision has been taken. It is this thought that leads to the main proposal of this article—the introduction of what we term provi- sional assessments. As explained more fully in section II, provisional assessments are short papers fo- cused on a current policy issue and delivered to the policy maker early in the research process. With timeliness at a premium, the preliminary analysis presented in these submissions should be based entirely on existing, easily accessible information and routine calculations, with evidence from other countries or extrapolation of historical trends being called upon to supplement whatever hard data are at hand. In consequence, a provisional assessment should also indicate whether its findings are sufficiently robust to warrant an immediate decision, and if not, what additional evidence and analysis are needed. Ideally, the latter conclusion should lead to a request from the policy maker for a more thorough investigation, accompanied by an indication of when results are required if they are to be useful. If, as might first appear, provisional assessments simply provide an additional arrow in the re- searcher’s quiver to be used when and where needed in the effort to inform policy, then, apart from a few remarks on implementation, no further discussion would be required. Provisional assessments come, however, with a potential cost. The danger is that the policy maker acts immediately on receiving one. Had the researcher bided her time, completed her research as thoroughly as possible, and only then sub- mitted a final report, its findings might have pointed to a quite different policy. Meanwhile, the policy maker, who is buffeted by the pressures of office, might have made his decision well before the final The World Bank Economic Review 555 report reaches his desk. In short, there is a trade-off between timeliness and rigor. Section III addresses this trade-off analytically. It is one of the first attempts to our knowledge to go beyond the reliance, in the research-policy literature, on case studies by offering a rigorous analytical framework. It is demon- strated that a provisional assessment, backed by the promise of a more complete assessment to follow if the provisional one points to significant uncertainty about the policy’s social profitability, is superior to Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 the current practice of submitting only a final report. Section IV deals with implementation. It explores ways of promoting the use of provisional submissions, currently a missing component in the research-policy chain, initially by means of inexpensive, donor-funded subsidies. It also proposes checks on provisional assessments in order to ensure quality and reduce the chan- ces of an incorrect decision, even if that decision is taken immediately on receipt of the preliminary analysis. The section also discusses ways of increasing the likelihood that the initial approach to the policy maker does in fact lead to whatever further research is recommended in the provisional assessment. These practical mea- sures buttress our claim that provisional assessments, if widely used, could have a significant and favorable impact on policy in many situations. Section V summarizes the argument and proposals. II. Provisional Assessments Rigor Reigns The primacy of rigor over timeliness is not surprising given that the incentives prevailing within the eco- nomics profession promote rigor before all else. There are well-established mechanisms to ensure that professional standards are met through the widespread use of peer reviews; and researchers, who natu- rally seek professional advancement, are keen to submit their research to a thorough, independent vet- ting by other scholars. There is, in contrast, no such history of individual or collective action from within the profession to promote timeliness in relation to policy making. As one illustration, consider the widely held view that randomized controlled experiments are supe- rior to other techniques of impact evaluation such as multivariate regression and propensity score match- ing, even though these other options offer the prospect of yielding results more quickly. Thus, the website of the Abdul Latif Jameel Poverty Action Lab notes that “Randomized evaluations are often deemed the gold standard of impact evaluation, because they consistently produce the most accurate re- sults” (http://www.povertyactionlab.org/about-j-pal).1 To compound matters, the instruments currently used to reach policy makers—the media, dedicated sem- inars, and policy briefs2—are not well designed to ensure timeliness, since they are almost invariably brought into play only after the research has been completed and reviewed. Thus, the main vehicles now used to strengthen the impact of research on policy reinforce, or at least coincide with, the researcher’s commitment to rigor, in that both imply an interaction with the policy-maker that follows research, and therefore raise the question of whether as much as possible is being done to ensure the timely delivery of results. A Way Forward The experience of the International Development Research Centre (IDRC), perhaps the leading interna- tional organization as far as linking research and policy is concerned, as well as that of other institutions, suggests a way forward. In particular, Carden (2009, 45), drawing on a detailed review of 23 IDRC- funded research projects and other evidence, stresses two points: first, the importance of initiating a 1 Similar statements abound: the mission statement of the Coalition for Evidence-Based Policy asserts that “evidence of effectiveness generally cannot be considered definitive without ultimate confirmation in well-conducted randomized controlled trials” (http://coalition4evidence.org/). The Report of the Evaluation Gap Working Group observes that “most questions about the impact of social programs require collecting data over years. Valid evidence of a program’s effectiveness often cannot be produced in less time” (Savedoff et al. 2006, 25). 2 For a useful overview of policy briefs, see Jones and Walsh (2006). 556 Bell and Squire dialogue with the policy maker as early as possible in the research process; and second, the value of sub- mitting short papers tightly focused on the policy question under consideration, as opposed to the more traditional, methodology-heavy accounts typically produced at the end of a research project. It is this combination of early interaction and policy-focused submissions that leads to the main proposal of this article—the introduction of what we term provisional assessments. Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 The type of research for which provisional assessments are intended is typically country-specific, is- sue-specific, and employs techniques such as cost-benefit analysis, cost-effectiveness analysis, and impact evaluations, possibly supported by econometrics or partial and even general equilibrium models.3 The following case study provides an example of how a provisional assessment can be used to address the type of well-defined policy question we have in mind. The issue considered by Annor-Amevor et al. (2012) is the low pass rate for the Basic Education Certificate Examination achieved by Ghanaian children in junior high schools, a matter of then current con- cern in Ghana.4 In particular, the authors assess the merits of participatory remedial classes for those who have failed the examination, combined with an opportunity to re-sit without repeating previous school grades. Besides useful background information, the study provides information on four key magnitudes: • Size of the population likely to be affected. The authors estimate that 41,686 children failed the exam- ination in the five regions under study. • Costs. They compile a detailed estimate of expected costs, concluding that the cost per pupil will be $254.50. • Impact. Based on experience elsewhere, they assume that the pass rate will increase from 62 to 77 per- cent as a result of the intervention. • Benefits. Using current Ghanaian data on earnings, they project that a certificate-holder will earn 20 percent more than someone without the certificate. Armed with this information, the authors conduct a cost-benefit analysis and arrive at a positive net present value for the proposed intervention. They also undertake a sensitivity analysis, especially with re- spect to impact and benefits, which suggests that the initiative is worth undertaking for a wide range of estimates. The example highlights the three characteristics that are likely to be typical of provisional as- sessments: focus on a well-specified policy issue; use of standard techniques; and reliance on readily available information. The Timeliness-Rigor Trade-off The Ghanaian example illustrates the trade-off between timeliness and rigor. All required information for this analysis could be obtained from existing sources because the analysts were willing to draw not only on “hard” evidence (coverage and costs) but also on the “soft” kind, in the form of international experience (impact) and extrapolation (benefits). This emphasis on ready availability should not be un- derstood to rule out the possibility of collecting original data; but the key point is that the information underlying a provisional assessment, whatever the source, has to be marshalled quickly.5 Even though 3 Another kind of research attempts to improve our understanding of development in the broad. The findings of such re- search typically enter the policy-making process indirectly and only gradually. They first appear in, and then percolate through, the pages of academic journals, those important devices for quality control. Only then do they enter policy fora and permeate policy discussions. 4 This is one of twelve country studies in a research project entitled “Strengthening Institutions to Improve Public Expenditure Accountability,” managed by the Global Development Network and Results for Development. 5 An example involving the use of specially collected data is provided by an analysis of the extension of the Nicaraguan Social Security Institute’s health insurance program to informal sector workers. The easily ascertained fact that the pro- posed cost of the insurance was considerably more than current, out-of-pocket health expenditures clearly pointed to the likelihood of low take-up rates, an indication subsequently confirmed by a controlled randomization experiment (Thorton et al. 2010). The World Bank Economic Review 557 this approach will usually entail reliance on data that are soft and potentially unreliable, professional in- tegrity is in no way compromised. All that is required is that the provisional assessment states clearly which inputs into the analysis have been established fairly accurately, which have been based on histori- cal or international experience,6 and how use of the latter affects the robustness of the findings. In the Ghanaian example, the robustness checks suggest that the downside risk of sacrificing full rigor Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 is negligible, but this cannot hold in general. In evaluating the potential of provisional assessments, therefore, it would be desirable to have some sense of the size of this downside risk and, going a step fur- ther, whether the sacrifice of rigor involved in providing such assessments likely nullifies their advantage in terms of timeliness, a task to which we now turn. III. Analyzing the Trade-off A project or program is under consideration. Both the exact timing of the simple “yes-no” decision of whether to adopt it and its social profitability are uncertain. There are two actors, a policy maker and a researcher, hereinafter denoted by D and R. As “insiders” in their respective fields, the former is better informed about timing, the latter about social profitability, but neither is perfectly so. An exchange of in- formation, however incomplete or indirect, may improve the chances that the decision, when it does come, will be the right one, in the sense that the chosen course of action is welfare-superior when viewed ex ante at the time of decision. There is always an alternative to the action in question, and consequently a cost of choosing wrongly. The net social benefit of making the correct decision, when it occurs, is assumed to be given, and is there- fore omitted from the analysis. In this connection, it should be recalled that otherwise identical projects or policies initiated at different dates are indeed different. In general, therefore, timing will affect the level of social profitability ex ante as well as an outcome ex post. Both parties are assumed to be aware of this fact, though R will be able to formulate it more exactly when assessing the probability that an af- firmative decision at a particular time t will be correct. An assumption about how to treat uncertain outcomes in public decision-making is also needed. Let the assumptions for the validity of the Arrow-Lind (1971) theorem hold, namely, that the project’s net returns be evenly spread over a large population and statistically independent of those of other projects. It is then valid to use expected values in making decisions, even though some outcomes are poor when viewed ex post. For simplicity, we also omit the social cost of producing the advice from the formal analysis.7 Let the variate T denote the date on which the decision occurs. At the outset (at time t ¼ 0), D’s sub- jective prior is that T has the continuous distribution function H(t), whose support is ½t1 ; t2 Š. In the ab- sence of the specific advice which will be treated below, his own knowledge, understanding, predilections, and such other opinion—expert or otherwise—at his disposal combine to yield the proba- bility qt that the decision, if it occurs on the date t, will be the correct one. Along with political pressures, these various influences are themselves likely to fluctuate over time. This does not, however, imply that, for any given t, qt is “fuzzy.” In this, we follow Elga (2010), who argues that rational agents must have sharp subjective prior probabilities: for each and every t 2 ½t1 ; t2 Š, D has an exact ex ante qt 2 ½0; 1Š. At 6 Two recent articles address issues surrounding the use of estimates of impacts drawn from other countries. Dhaliwal et al. (2012) provide guidance on how to compare results based on RCTs, using evaluations of educational programs in multiple countries as an example. Pritchett and Sandefur (2013) add the important point that context matters, arguing that experimental results from the right context are currently a better guide to policy than nonexperimental ones from a different context. 7 To a fairly good approximation, this is the value of the researcher’s time devoted to the task when priced at her shadow wage rate. 558 Bell and Squire the outset, therefore, D assesses the corresponding ex ante probability that the decision will be correct to be ð t2 Q¼ qt dHðtÞ: (1) Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 t1 Aware that the issue is on D’s agenda but initially uninformed about his H(t) and qt , R undertakes to an- alyze it.8 The longer she devotes herself to this task, the better founded the advice will be, and the more compelling will be the way in which she frames the evidence and arguments. The advice will therefore improve as time passes, especially in the sense of increasing the probability that the correct decision will be taken—provided the memorandum is actually on D’s desk when the decision point arrives. Timeliness, then, is essential if her efforts to make a persuasive case are not to be in vain. When weighing all these considerations, she has her own subjective priors at the outset, F(t) and pt , respectively. Even if the research results are delivered in time, whether they are used depends on D’s response. In particular, it is arguable that the gap between the memorandum’s arrival and the actual decision matters. Advice that arrives far in advance may suffer a loss of effectiveness simply because it appears a bit dated when that moment of decision does arrive. There is even the mundane possibility that the memorandum will get mislaid or overlooked, as D’s office struggles to deal with the stream of matters demanding his attention. Working against these hazards is the fact that a memorandum’s early arrival gives him the op- portunity to reflect on it, which should make it more influential. D and R face a common hazard, namely, that there will be a “changing of the guard” before the deci- sion actually occurs.9 The new regime may take the form of a new government or, if D is a senior bu- reaucrat, his replacement through regular rotation or otherwise. As D and R reflect on the possible timing of the decision and the associated probability that it will be correct, each must also form a prior probability of whether a changing of the guard will take place, and if so, when and with what conse- quences for timing and a happy outcome. There are three broad possibilities, though D and R may well differ in their assessments of whether, when and the resulting consequences. First, there is a firm conviction that the decision will occur before any changing of the guard. Second, if such a change could happen, its date can still be forecast without error, as might well be the case in a stable democracy with a fixed electoral cycle and any stable, compar- atively independent bureaucracy. Then the forms of H(t) and qt will reflect D’s assessment of how the event will influence them, and the same applies to R’s F(t) and pt . Note that this includes the special case in which the said date precedes the prior estimate of the earliest possible date for a decision. The remain- der of this section is formulated on the basis that one or other of these two broad possibilities holds. Turning to the third possibility, what happens if a change of regime within the relevant time interval cannot be ruled out and its timing is uncertain? In principle, D must formulate a pair H(t) and qt for each date on which such a change can occur with positive probability, together with the associated prior probability that it will occur on the date in question. The resulting compound lottery will tax most peo- ple’s powers of analysis, so D searches for a plausible simplification. Assumption. The event in question occurs on the date td 2 ðt1 ; t2 Þ with probability pd , or not at all. If it occurs, H(t) is unaltered up to t ¼ td and is amended to H d ðtÞ over the remaining, revised interval ½td ; t2d Š; and qt is likewise amended. 8 We assume that the interaction between the researcher and the policy maker is initiated by the former, this being, in our view, the more realistic representation. The analytical apparatus could, of course, be re-jigged to make the policy maker the instigator, a desirable extension that we discuss, among others, in section IV. 9 We are indebted to a referee for urging us to examine this possibility. The World Bank Economic Review 559 Note that the problem only arises when td lies in the said interval; otherwise, either the first or second broad possibilities set out above will apply. The single possible date represents a sort of average, albeit one weighted by whatever information at his disposal. In a very uncertain setting, he might well choose the half- way point. Let the same assumption apply, mutatis mutandis, to R’s view of things. The Appendix sets out how the basic elements Q, and—see immediately below—R’s pt ðsÞ and PðsÞ must be reformulated to cover Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 this third possibility, whereby the extension to two or more possible dates is obvious. With these considerations in mind, let R’s prior probability that a memorandum submitted at time s will result in a correct decision occurring at time t be as follows: ( 1 À ð1 À pt Þ/ðs; t À sÞ > pt if 0 < s t; pt ðsÞ ¼ (2) pt otherwise; where the “influence function” / reduces 1 À pt , the probability of an erroneous decision. This function is assumed to be decreasing in its first argument (research quality), but it may be decreasing or increasing in the second (D’s response to the gap between the delivery of the advice and taking the decision). The assumption that a submission that arrives before or at the point of decision makes a correct decision more probable—but not certain—is plausible. For if R is competent, even her preliminary analysis is very likely, on average, to improve on whatever process yields D’s unassisted qt . This assumption implies that / < 1 if s > 0, with /ð0; tÞ ¼ 1. Observe also that the larger is pt , the smaller is the effect of a spe- cific piece of advice, pt ðsÞ being at most unity. A Single Submission Let R obtain no additional information about the timing of the policy decision in the course of her work, so that she gains nothing by postponing her decision about when to submit her memorandum, and she will hold fast to the plan s chosen at the very start. If she waits beyond t1 , it is possible that she will submit her advice too late. This must be weighed against the improvement in its quality that would result from taking longer over it. The plan s yields her ex ante probability that the policy decision will be the correct one: ð s t2 ð PðsÞ ¼ pt dFðtÞ þ 1 À ð1 À pt Þ/ðs; t À sÞ dFðtÞ t1 s (3) t2 ð t2 ð ¼ pt dFðtÞ þ ð1 À pt Þ½1 À /ðs; t À sފ dFðtÞ: t1 s The trade-off is described exactly by differentiating w.r.t. s: t2 ð 0 P ðsÞ ¼ Àð1 À ps Þ½1 À /ðs; 0ފ dFðsÞ À ð1 À pt Þð/1 À /2 Þ dFðtÞ; (4) s where /i denotes the partial derivative of / w.r.t. its ith argument. The first expression on the right-hand side is the loss in terms of P caused by spending a little more time on the memorandum, and so increasing the chances that it will arrive too late; the second expression is the corresponding gain arising from the im- proved analysis, if it arrives on time. As noted above, the assumption that advice becomes more influential as its quality improves implies that /1 < 0. While /2 may take either sign, it is very plausible that, on balance, better analysis will reduce the chances that the wrong decision will follow a timely submission, that is, d/ ¼ ð/1 À /2 Þ ds < 0. 560 Bell and Squire Let sà  arg max PðsÞ s2½t1 ;t2 Š denote R’s optimal plan. It is clear from equation (4) that waiting until the very last minute (s ¼ t2 Þ is not optimal. Thus, a researcher’s natural inclination to delay any policy input until all refinements have been in- Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 corporated, and all conceivable robustness checks undertaken, may very well defeat the purpose of influenc- ing the decision. Indeed, the possibility that, at the other extreme, taking no chances at all is optimal, that is, sà ¼ t1 , cannot be ruled out. Indeed, if /1 À /2 ! 0, then that is exactly what R should do. A Provisional Submission The preceding subsection sets out the costs and benefits of delaying a single submission when R cannot acquire additional information about the timing of the policy decision. We now allow her to test the wa- ters by submitting a provisional assessment, accompanied by the offer of a more definitive analysis to follow if required. This provisional assessment also goads D into playing an active role in the ‘game’ be- tween them. We shall prove that the resulting option has a positive value, relative to a single submission, in the sense of increasing the probability that D will decide correctly.10 Let R submit a provisional assessment at time s 2 ½t1 ; t2 Š. If the decision has already been taken, she is so informed, and there is nothing more to be done. If, on the contrary, the decision is still due, let D re- spond in just one of three ways. First, with her memorandum now lying on his desk, he waits no longer and makes his decision, and R is informed accordingly. It is rather likely that D’s precipitate action is prompted by the memorandum telling him what he wanted to hear, but its professional content still raises the probability that the decision is correct; for the memorandum reflects R’s assessment of the project’s social profitability, albeit a provisional one. On the basis of the information available to her, the probability that the decision is correct is 1 À ð1 À ps Þ/ðs; 0Þ, with a corresponding improvement over ps of ð1 À ps Þð1 À /ðs; 0ÞÞ. Second, D expresses a clear interest in a more definitive assessment and sets a deadline tà 2 ðs; t2 Š, which both parties regard as fixed, and the final memorandum is submitted at tà . This more measured response requires D to choose a deadline in the light of his subjective priors at t ¼ s concerning the tim- ing of the decision and the probability that it will be correct. His decision problem is therefore analogous to R’s in the case of a single submission. Let there be no new information about timing, other than the fact that the decision is still outstanding, so that H(t) is truncated at t ¼ s. Since R will have worked on her assessment throughout the period up to t ¼ tà , let its effect on D’s prior at t ¼ s that the decision, when it comes, will be correct take the form analogous to equation (2): ( à 1 À ð1 À qt Þwðtà ; t À tÃ Þ if s < tà t; qt ðt Þ ¼ (5) 1 À ð1 À qt Þwðs; t À sÞ otherwise; which allows for the possibility that events will force a decision before the deadline tà and so confine R’s influence on the proceedings to her provisional memorandum. Corresponding to R’s ex ante probability P, D’s Q at t ¼ s is given by tà ð à ½1 À H ðsފQðt ; sÞ ¼ ½1 À ð1 À qt Þwðs; t À sފ dHðtÞ s : (6) t2 ð þ ½1 À ð1 À qt Þwðtà ; t À tà ފ dH ðtÞ tà 10 The provisional assessment is most unlikely to pass muster with a refereed journal. We return to the issue of quality in section IV. The World Bank Economic Review 561 The trade-off confronting D when choosing tà so as to maximize Qðtà ; sÞ is also analogous to that fac- ing R in the case of a single submission and need not be set out in detail. Let tÃ0 denote the maximizer in question. Since D’s course of action necessarily involves tÃ0 > s, it follows that it must yield an improve- ment not only over receiving no advice from R at any stage but also over just her provisional assessment at t ¼ s. Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 R is privy only to D’s specified deadline. While she could infer something about D’s priors at t ¼ s, such speculation will serve little or no purpose in this round—though it might do in the future should they have further dealings. She therefore labours away until submitting her final assessment at tÃ0 . Having done so, she achieves a total improvement over her ex ante prior ptÃ0 of ð1 À ptÃ0 Þð1 À /ðtà ; 0ÞÞ, as she sees things. D’s third possible response to R’s provisional assessment is to send her a polite, noncommittal ac- knowledgement, so that she gains only the information that the decision is yet to come. She must now decide how much more time to invest in producing a final memorandum, with no firm deadline. It is the remaining uncertainty in the event of such an acknowledgement at s that makes the problem of back- ward induction, and hence of determining the optimal plan for s at t ¼ 0, nontrivial—in principle, at least. The corresponding optimal timing of her final submission, denoted by n0 ðsÞ, must be found in or- der to establish the optimal choice of s at t ¼ 0. She is now back in the setting of a single submission, with starting point t ¼ s. It is plausible that learning that the decision has yet to be made will have no effect on her priors other than truncating F(t) at t ¼ s. Let this be so. Then the effect of a provisional submission that draws a noncommittal response on the probability that the final decision will be correct will be given by the upper branch of equation (2), unless it is followed up by a timely final submission, in which event, n will replace s in /ðÁÞ and thus represent the improvement in the quality of the advice. Conditional on the event that such an acknowledgement is received at s, the choice n 2 ½s; t2 Š yields, analogously to equation (3), t2 ð ½1 À FðsފPðn; sÞ ¼ ½1 À ð1 À pt Þ/ðs; t À sފdFðtÞ s (7) t2 ð À ð1 À pt Þ½/ðn; t À nÞ À /ðs; t À sފdFðtÞ; n where n0 ðsÞ  arg max Pðn; sÞ: n2½s;t2 Š The associated first-order condition for an interior maximum is analogous to equation (4): t2 ð 0 Àð1 À pt Þ½/ðs; n À sÞ À /ðn; 0ފF ðnÞ À ð/1 ðn; t À nÞ À /2 ðn; t À nÞÞdFðtÞ ¼ 0: (8) n A comparison of the two conditions establishes that n0 ðsÞ is indeed dependent on s when the latter is the date on which the provisional assessment is submitted. To be more precise, consider the choice n ¼ s < t2 , which is effectively the same as making a single, final submission at s. The first term in brackets on the right-hand side of equation (8) vanishes, and the integral is negative, in virtue of /1 À /2 < 0. Hence, n0 ðsÞ > s, and the difference between the values yielded by submitting at n0 ðsÞ and s, respectively, yields the (positive) option value of a provisional submission for any choice of s 2 ðt1 ; t2 Þ, conditional on R receiving a noncommittal reply. 562 Bell and Squire The size of the option value varies, in particular, with s and the behavior of /. Inspection of equation (7) reveals that the option value will be small if s is sufficiently close to t2 . For a delay in making the pro- visional submission reduces the time in which to make the most of the resulting option; and it lowers the probability that the option will arise in the first place. It is also seen that the option value is smaller, cet. par., if / is decreasing in its second argument; for the interruption at the point n reduces the delay from t Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 Às to t À n thereafter. If, conversely, / increases with the delay, the interruption will be advantageous. Now R’s choice of s also arguably affects the probabilities of getting the responses listed above, con- ditional on the decision yet to be taken. That is to say, the act of making a provisional submission also affects the timing of the decision whose outcome it intends to influence. It is plausible that the closer s approaches the upper limit t2 , the greater is the chance that it will “trigger” the decision, an outcome whose probability is denoted by p1 ðsÞ. The opposite holds for the probability of getting a noncommittal response, p3 ðsÞ; for the earlier the provisional submission is made, the sketchier it is likely to be—and to appear. At all events, R’s plan to submit a provisional assessment at s, with her rational anticipation of choosing n0 ðsÞ in the event of receiving a noncommittal acknowledgement, involves the following possi- ble outcomes: (i) the provisional memorandum arrives too late, with probability FðsÞ; and (ii) it arrives before the decision, with probability 1 À FðsÞ, and then elicits one of the three said responses from D, P 3 with respective probabilities pi ðsÞ, i ¼ 1; 2; 3, pi ðsÞ ¼ 1. i¼1 There is, however, one further complication to be resolved before R chooses s at the outset, namely, that the contingent deadline tÃ0 has yet to be revealed. The latter certainly depends on s. R knows noth- ing about D’s priors at this stage, so it is natural for her to assume that this date, denoted by te , is uni- formly distributed on the interval ½s; t2 Š. Yet whatever be her prior distribution of te in relation to s, R’s ex ante prior probability that the decision, when it does come, will be correct is ðs Pðs; n0 ðsÞÞ ¼ pt dFðtÞ þ ½1 À Fðsފ  fp1 ðsÞð1 À ð1 À ps Þ/ðs; 0ÞÞ þ p2 ðsÞE½ð1 À ð1 À pte Þ/ðte ; 0Þފ þ p3 ðsÞPðn0 ; sÞg; t1 (9) where E is the expectation operator. Inspection of equation (9) reveals that, as R sees things, the proba- bility of securing a correct decision with a provisional submission is necessarily superior to that with a single, final submission if p1 ðsÞ < 1. For a single submission is equivalent to declining an invitation to submit a final assessment against a fixed deadline, or, in the event of receiving only a noncommittal re- ply, to choosing n ¼ s whatever be the choice of s, and it has been proved above that neither course of action is ever optimal. The choice of s itself is influenced by various considerations. Under the above assumptions, a provisional submission before t2 always yields, with strictly positive probability, the opportunity both to deliver the (con- strained) best advice and to adjust the plan optimally when uncertainty about the timing of D’s decision still remains. Against this, there is the drawback that a provisional submission will, with positive probability, pre- cipitate an immediate decision. For its arrival may focus his attention on the matter in question, especially if there is a good chance that a leak might arouse the interest of other influential parties—and provide them with ammunition. If /ðÁÞ is decreasing in its second argument, such an early triggering of the decision will de- crease the probability that it will be the right one. For the delay t À s between the provisional submission and the decision then has a favorable effect on / – whenever there is such a delay. The above proof of the superiority of a provisional over a single, final submission implies that the trigger effect cannot, on average, outweigh the others.11 To grasp why this is so, consider the limiting 11 In particular instances, the decision will indeed be triggered, with an attendant adverse effect; but the claim of superi- ority relates to a whole series of “trials.” The appeal to the Arrow-Lind theorem is important here. The World Bank Economic Review 563 case in which the trigger effect occurs with certainty ðp1 ¼ 1; p2 ¼ p3 ¼ 0Þ for all s; for then there is ef- fectively neither an option nor an invitation with a fixed deadline, and hence no advantage to a provi- sional submission. Indeed, all provisional submissions become final ones. There are, of course, consequences for the optimal choice of s, but that is another matter. In this connection, it should also be noted that in the case of a single submission, there is neither a trigger effect nor even the interim informa- Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 tion as to whether the decision has been taken. In equation (9), the trigger effect is expressed by the term involving /ðs; 0Þ, but the corresponding term in equation (3), /ðs; t À sÞ, contains the delay t À s and some delay is virtually certain. The two models are not therefore fully comparable, and the intuition af- forded by the one cannot be applied wholesale to the other. While the potential triggering effect of a provisional submission cannot overturn the main result of this section, it can greatly reduce the superiority of the plan ðs; n0 ðsÞÞ over a single, final submission. Observe that the option value arises from the second and third terms within braces in equation (9). Now suppose D’s behavior tends to be of the hair-trigger kind, so that the triggering effect approaches a near certainty even if R attempts to keep it at bay by making a provisional submission soon after t1 . Since the remaining terms within braces on the right-hand side of equation (9) approach zero as p1 approaches unity, the claim follows. More generally, all of the above considerations duly appear in the first-order condition for Pðs; n0 ðsÞÞ to take a maximum, whose baroque details need not be laid out here.12 Suffice it to say that the said con- dition, together with condition (8), characterizes the optimal plan ðs0 ; n0 ðs0 ÞÞ. Models of this kind are best thought of as a means of ordering one’s ideas through a private con- versation, with the model as partner. No one in her right mind is going to set about calculating the optimal plan ðs0 ; n0 ðsÞÞ on the lines set out above: researchers have better things to do with their time. Yet the analysis points to conclusions of practical importance. In order to exploit fully the ben- efits of a provisional submission, and to minimize the danger of incorrect decisions, researchers need an appropriate framework of incentives, policy makers (and donors) need an adequate quality- control mechanism, and both parties need some mechanism for encouraging communication. Institutional support in these areas would surely help all parties to allocate their time and energies more efficiently where promoting better decisions is concerned. These themes will be pursued in the next section. IV. Provisional Assessments in Practice The first part of section III demonstrated formally the rather obvious, albeit routinely ignored, point that delaying a single submission until the underlying research has been refined to the fullest possible extent is unlikely to be a wise strategy, since there are very good chances that the policy maker will have acted before the advice arrives. If this happens, his decision will not be informed by any specific guidance from the research community, and the resources devoted to the research will be wasted—at least from an im- mediate policy-making perspective. In these circumstances, an alternative approach, in the form of an early, provisional submission with the promise of a more refined analysis to follow, may well be prefera- ble. This thought led to the analysis in the second part of section III and thence to the key result that the said alternative is in general superior to a single, final submission. Several factors caution against concluding that this result points to a simple way of improving the impact of research on policy. First, as noted in the Introduction, the research-policy interaction is a messy business and decidedly context-specific. The analytical framework presented above necessar- ily abstracts from this complexity and captures only selected aspects of reality. Second, and also noted in the Introduction, the incentives currently confronting researchers deter provisional 12 They are available from the authors on request. 564 Bell and Squire submissions and favor more polished outputs aimed at refereed journals. Third, the key result—that a provisional submission, coupled with the offer of a more thorough analysis, is superior to a single submission—holds only in a probabilistic sense: it is correct on average over a long run of trials, but not necessarily in each case. We address these three points in turn. We begin with some comments on the analytical framework’s value added and its applicability in a wider range of circumstances Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 than may be apparent at first sight. We then discuss how incentives may be changed to promote greater reliance on provisional assessments. Finally, we outline some measures to increase the likeli- hood of a positive outcome and counter the risk of an early and incorrect decision based on faulty analysis and evidence. Value Added and Applicability The model in section III was designed specifically to illustrate the trade-off between timeliness and rigor and to allow an exploration of the merits of provisional assessments. The intention was to cut away the tangle of real-world complexity and context-specificity to expose the essential tension be- tween meeting professional research standards and serving policy makers in a timely manner. This analytical approach departs from the reliance on case studies that has to date dominated research on the research-policy link. The strength of case studies lies in their ability to delve into the rich detail of specific examples, but they are not easily generalizable and usually lack analytical rigor. The model presented above builds on, and brings analytical confirmation to, the results of existing, case- specific research by demonstrating analytically the general merit of two findings that have emerged from observation in a variety of different circumstances: first, successful research-policy episodes are often characterized by early interaction between researcher and policy maker; and second, short, fo- cused inputs are usually the most effective means of communication. The value added of the model lies primarily in its rigorous demonstration that these two elements, when combined in what we have called provisional assessments, do indeed yield outcomes superior on average to those yielded by the traditional approach of waiting until the research is complete in all respects before sharing re- sults with policy makers. The framework also offers an approach to thinking about the interaction between policy-makers and researchers in an orderly way. For it is flexible enough to be more widely applicable than its formal structure might at first suggest. For example, the initial exchange between researcher and policy maker could lead to several rounds during the decision process rather than the single exchange assumed in the model. Likewise, the researcher-led version in section III could be reversed to allow for an opening ap- proach from the policy maker. Or the provisional assessment could be initiated after the main research study has begun if there is still scope for adjusting the delivery date of its results.13 There is a place for other agents, too. Thus, if the policy maker has little interest in research and ignores the provisional as- sessment, the researcher could make her preliminary results available to the media and NGOs in the hope of influencing the decision by arousing public opinion. It should also be remarked that the nature of the decision to be taken, as formulated in section III, is not especially restrictive. Instead of a straightforward yes-no decision on a single project or program, the set of choices can be extended to an array of specific, competing alternatives, in addition to the unspeci- fied alternative, in any cost-benefit analysis that corresponds to rejecting the single proposal if its net pre- sent value at shadow prices is negative. To give examples, the tendering process often yields different designs of the same project, be it a port or a primary school. Likewise, a policy intervention may allow different sequences ex ante. The timing of the analysis can also vary. Thus, a program or policy that 13 In the event that the provisional assessment convincingly questions the merit of the program or policy, then the re- search study should be abandoned. This is a risk, but the common-sense message here is—undertake a provisional as- sessment before embarking on a major research exercise. The World Bank Economic Review 565 stalls during implementation may present the policy maker with a cancel-or-redesign decision that could benefit from a provisional assessment.14 Similarly, but earlier in the project cycle, the analysis could be applied to a pilot project. Indeed, any specific decision that can be subjected to empirical analysis involv- ing standard economic tools is a possible candidate. Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 Leveling the Playing Field As argued in the Introduction, the ruling incentives within the economics profession encourage re- searchers to refine and verify their analyses as fully as possible before sharing their results with policy makers. It follows that if the aim is now to promote greater use of provisional assessments, it will be nec- essary to modify researchers’ behavior by means of new incentives specifically designed to counter exist- ing ones. The obvious way of leveling the playing field is to subsidize provisional assessments. Yet it is unlikely that policy makers in developing countries would be willing to pay for these before a solid track-record has been established. Consequently, a third party—the obvious candidate is the international donor com- munity, which is already an important funder of research in developing countries—has to be the source of the subsidy, at least initially. The payment per submission would have to motivate a significant num- ber of researchers to participate and would therefore have to exceed the marginal cost of delivering the input. The cost of producing a preliminary assessment, given its nature, is likely to be small. Even with some additional premium to guarantee adequate participation, therefore, the total cost of the subsidy would remain modest. The appropriate size of this premium could be determined through experimenta- tion, with donors adjusting the amount until sufficient numbers of provisional assessments are tendered. To ensure that the subsidy does indeed result in an early submission to the policy maker, donors could undertake two specific actions. First, they could process requests for funding, and disburse funds, quickly. Second, the initial grant could be limited to the costs of the study, with the premium paid only on confirmation that the provisional assessment has been delivered to the policy maker within some rea- sonable, and precisely specified, time frame. Additionally, and to encourage interaction between researcher and policymaker, donors could give priority to those proposals for which the researcher can produce a written endorsement from the relevant policy maker15 or where the policy maker initiates the process himself. Such expressions of interest would demonstrate both that contact between the parties has been established and that the policy maker can be expected to act on the analysis when it arrives. In some cases, a policy analyst inside a ministry may undertake the preliminary research for a provisional assessment. This attractive practice could be encouraged by extending the subsidy to government analysts wherever legal rules and bureaucratic pro- cedures allow. Whatever the origin of a research-policy interaction, if policy makers share their timetable with re- searchers following receipt of a provisional assessment, then the likelihood that any ensuing final sub- missions will arrive on time should increase significantly. Donors should be especially interested in promoting this outcome, since it ensures that research output at a more refined stage is actually feeding into the policy-making process. This highly desirable result could be encouraged by declaring that re- quests to fund full-scale research projects will receive preferential treatment if they have been preceded by a provisional assessment. The exact advantage—speedier turn-around or more generous funding— could be decided by individual donors; but whatever its form, the knowledge that funds are more likely to be forthcoming for a fully fledged research proposal provided a provisional assessment has been 14 Variants of the project in question may suggest themselves, whether from experience or even institutionalized learning in the course of execution itself, as advocated vigorously by Pritchett et al. (2012). 15 The International Initiative for Impact Evaluation uses this approach. Researchers are asked to include in any applica- tion for funds an endorsement from the relevant policy maker. 566 Bell and Squire completed, will serve as a further and powerful inducement for researchers to interact with policy mak- ers early in the research cycle. Since time is of the essence, the administrative procedures for funding both provisional assessments and full proposals that have been preceded by a provisional assessment should be streamlined, with quick turn-around for the former and almost automatic funding for the latter. This departure from the Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 thorough vetting of full proposals should be balanced by a systematic effort to monitor each stage of the research-policy interaction from start to finish. This would provide reassurance that, simplified proce- dures notwithstanding, the approach is yielding the desired result in terms of the frequency with which policy makers move from provisional assessments as a first step, to final submissions as a contingent sec- ond step, and then to policy action, or otherwise indicate the need for fine-tuning. An attractive side- benefit of this monitoring program is that the information so generated would create a unique data base capable of expanding and deepening the profession’s understanding of the research-policy interaction well beyond that afforded by sporadic case studies. Reducing the Risk of Errors The introduction of provisional assessments would improve policy making in the large; but, in specific cases, the advice may increase the probability that the wrong decision is made. The initial analysis, pre- cisely because it is a first cut, may be so faulty as to point the policy maker in the wrong direction, whereas the alternative—waiting for the full analysis—would provide better guidance (assuming it ar- rives in time to be of use). The quality of provisional assessments is, therefore, an issue to be addressed. Confidence in quality can be increased by means of an independent, professional review of each pro- visional assessment prior to its delivery to the policy maker.16 Thus, researchers receiving grants to con- duct preliminary assessments could be required to submit their assessments to a review covering such basic aspects of their analysis as specification of the policy question, use of data, choice of analytical tool, and robustness checks. The reviewer, who should be of another nationality and drawn from a panel of experts, could also offer more general advice—on alternative designs, for example—drawn from both the published and the so-called grey international literature dealing with whatever program or policy is being evaluated. Making the premium to be paid to the researcher conditional on such a review would reduce the chances of unsatisfactory advice reaching the policy maker and adversely affecting policy decisions. The drawback with this suggestion is, of course, that reviews inevitably take time, and yet timeliness is essential if provisional assessments are to fulfill their intended purpose. To minimize any hold-up, re- viewers could be paid a fee per review, provided it is delivered within a specified time. The academic community, drawing on its well-established expertise in controlling quality by means of double-blind refereeing prior to publication, is well placed to contribute to this more specialized review process. Furthermore, compared with submissions to professional journals, provisional assessments are short, fo- cused pieces and should therefore admit of a speedy review. It is noteworthy that there are journals in other fields, such as medicine and branches of the natural sciences, that enforce a very rapid review, with accepted articles appearing in a matter of weeks after submission.17 Such journals devoted to policy questions would serve our present purposes admirably; 16 This point is not confined to provisional assessments; it applies equally to all existing means of reaching the policy- maker (e.g., policy briefs), which are often delivered without review. In their effort to reach policy-makers, many de- velopment specialists fail to subject research-based advice to careful scrutiny before delivering it, when in fact, such re- search should probably receive more scrutiny than research submitted to a professional journal. Weiss, in her foreword to Carden (2009), makes this point when she rightly queries the latter’s apparent belief that “research is al- ways good and right.” 17 We are indebted to a referee for drawing this alternative to our attention. The World Bank Economic Review 567 but there are none, and there seems little prospect of any unaided births of this kind. Donors might wish to act as midwives, but editors would have to be sure of keeping them at arm’s length subsequently. Even if adequate quality is assured, the pitfalls do not end with the submission of a provisional assess- ment. A policy maker may simply ignore its analysis and instead use its arrival as an excuse to move for- ward with a long-preferred, but quite possibly incorrect, decision. This is the worst sort of trigger effect: Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 the decision is based on personal experience, a yearning for advancement, political pressure, or other fac- tors quite unconnected with the research findings in question. One effective way of reducing this risk is to distinguish a priori those policy makers who are most likely to use a provisional assessment on its merits from those who are not. The best, and perhaps only sensible, solution to this problem is to leave the tactical details to local re- searchers, who are far better acquainted with local policy makers’ inclinations and behavior. These re- searchers will have the model and the option of submitting a provisional assessment in the back of their minds, but how they actually conduct the business will depend on the particular policy maker in ques- tion. Given her unique knowledge, the local researcher is the one best placed to identify those policy makers most likely to act prudently upon receipt of a provisional assessment. Having done so, her knowledge will be valuable at the next stage, if a more refined analysis is needed. For in practice, she will not be in the corset of the highly asymmetrically held information posited in sec- tion III. Rather she will likely have a prior, however fuzzy, regarding the form of what is termed the pol- icy maker’s “influence function” (the wðÁÞ of section III) and even some feeling for his view of the timing and quality of the decision (the H(t) and qt functions). These, too, should improve the probability of a correct decision. The requirement that a researcher seek the relevant policy-maker’s endorsement before receiving the premium will also sharpen her incentives. In effect, such a requirement functions as a method of “in- formed targeting,” one that relies on local-researchers’ knowledge regarding which local policy makers to approach and how best to conduct business with them if asked. In like manner, only more so, interac- tions initiated by policy makers or policy analysts can be thought of as stemming from a form of self- selection that can be expected to lead more often than not to better decisions. It is difficult to know how effective these methods of targeting may be; but even if they fail on occasion, the researcher always has recourse to the media and public opinion to exert pressure on politicians who misuse or ignore the find- ings of research. As noted in section III, the possibility of a changing of the guard will be on both parties’ minds, and it will surely crop up in any discussion between them. An attraction of our proposed scheme is that it pro- vides some measure of protection against this particular hazard in the policy-making process. If they see eye to eye on the likelihood of this event and it is not on the distant horizon, there will be an additional incentive to deliver a provisional assessment in short order, in the hope of tying the hands of any new re- gime, however the assessment turns out. Yet this potential advantage comes with a drawback; for if R senses that her view of the social profitability of the project is likely to be rather different from D’s, she may delay matters, precisely in the hope of a changing of the guard. Such strategic behavior by both par- ties cannot be ruled out. V. Concluding Remarks Researchers, policy makers, and donors are constantly seeking ways to improve the impact of research on policy. This article offers a practical way of contributing to this goal by encouraging the delivery of more timely inputs to the decision maker in the form of provisional assessments. This proposal draws on two observations emerging from the considerable case-study material now available on the interaction between research and policy. The evidence suggests that early interaction and short, policy-focused in- puts are often associated with successful outcomes. Our call for a much greater reliance on provisional 568 Bell and Squire assessments builds on, and is fully consistent with, these observations. By establishing analytically the su- periority, on average, of a provisional assessment, accompanied by the offer of a more refined analysis, over a single, final submission, this article strengthens the argument in favor of early and focused policy inputs. In this instance, the happy outcome is that both case-study evidence and analytical rigor point in the same direction. Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 The second, related contribution of the article is to address the chief difficulties of implementing such a scheme. Provisional assessments are not widely used at present, so there is little experience to go on. It is argued that prevailing incentives are the main cause, and that inexpensive donor-funded subsidies are the right remedy to get things started. Ensuring quality is a related concern. To deal with it, the paper calls for a fast but rigorous independent review of all provisional assessments and the tying of funds for more thorough research proposals to the timely submission of a related provisional assessment. The reshaping of incentives coupled with quality control can be thought of as supply-side measures that should encourage more reliance on provisional assessments, as a demand-side response. This, we have argued, will improve the quality of decisions in the public sphere. VI. Appendix It is assumed that on date td , and no other, there may be a changing of the guard, and then with probability pd . Under this assumption, equation (1) in the text becomes td ð t2 ð 2d tð Q¼ qt dHðtÞ þ ð1 À pd Þ qt dHðtÞ þ pd qr d t dH ðtÞ: (10) t1 td td If D has a poor opinion of the likely new regime or his own successor, qd t will be markedly smaller than qt . R revises her priors F(t) and pt in analogous fashion. Note that her priors tr and pr can also differ from D’s. If the event in question occurs, equation (2) in the text becomes ( 1 À ð1 À pr t Þ/ðs; t À sÞ > pt if tr < s t; pr t ð sÞ ¼ (11) r pt otherwise: Equation (3) now takes the form, compactly expressed in terms of pt ðsÞ and pr t ðsÞ, 2 2r 2r 3 ð t ð t tð 6 7 PðsÞ ¼ ð1 À pr Þ pt ðsÞdFðtÞ þ pr 4 pt ðsÞdFðtÞ þ pr r t ðsÞdF ðtÞ5: (12) t1 t1 tr Note that substitution for pt ðsÞ and pr t ðsÞ from equations (2) and (11), respectively, will yield PðsÞ in extensive form. Like D, R will have her own views about how a changing of the guard will affect the timing and correctness of the decision. References Annor-Amevor, E. A., and C. E. Afudego. 2012. “Simulating Alternatives for Increasing Basic Education Certificate Examination Pass Rate in Junior High Schools in Ghana,” mimeo, Integrated Social Development Centre, Ghana. Arrow, K. J., and R.C. Lind. 1971. “Uncertainty and the Value of Public Investment Decisions,” American Economic Review, 60: 364–78. Cabinet Office. 1999a. “Modernising Government,” The Stationery Office, London. Cabinet Office. 1999b. “Professional Policy Making for the Twenty First Century,” Report by Strategic Policy Making Team, London. Carden, F. 2009. Knowledge to Policy: Making the Most of Development Research, Sage Publications India Pvt. Ltd, New Delhi. The World Bank Economic Review 569 Court, J., I. Hovland, and J. Young. 2005. Bridging Research and Policy in Development: Evidence and the Change Process. London: ITDG. Elga, A. 2010. “Subjective Probabilities Should be Sharp.” Philosopher’s Imprint, 10 (5): 1–11. Dhaliwal, I., E. Duflo, R. Glennerster, and C. Tulloch. 2012. “Comparative Cost-Effectiveness Analysis to Inform Policy in Developing Countries: A General Framework with Applications for Education,” Conference Paper, Abdul Downloaded from https://academic.oup.com/wber/article-abstract/31/2/553/2897315 by Joint Bank-Fund Library user on 08 August 2019 Latif Jameel Poverty Action Lab, MIT. Jones, A. M., L. Squire, and R. Thomas. 2010. “Evaluating Innovative Health Programs,” Health Economics 19, Supplement 1, September. Jones, N., and C. Walsh. 2006. “Policy Briefs as a Communication for Development Research,” Overseas Development Institute, London. Jones, N., H. Jones, and C. Walsh. 2008. “Political Science? Strengthening Science-Policy Dialogue in Developing Countries,” Overseas Development Institute, London. Livny, E., A. Mehendale, and A. Vanags. 2006. “Bridging the Research Policy Divide in Developing and Transition Countries: Analytical Lessons and Proposals for Action,” Global Development Network, mimeograph, November. Nutley, S, M., I. Walter, and H. T. O. Davies. 2007. Using Evidence: How Research Can Inform Public Services, The Policy Press, Bristol. Overseas Development Institute. 2004. “Bridging Research and Policy in International Development,” ODI Briefing Paper, London. Pawson, R., 2006. Evidence-Based Policy: A Realist Perspective, Sage Publications Ltd, London. Pritchett, L., S. Samji, and J. Hammer. 2012. “It’s All About MeE: Using Structured Experiential Learning (‘e’) to Crawl the Design Space,” Working Paper No. 2012/104, UNU-Wider Working Paper No. 2012/104, Helsinki. Pritchett, L., and J. Sandefur. 2013. “Context Matters for Size: Why External Validity Claims and Development Practice Don’t Mix,” Working Paper 336, Center For Global Development, Washington, DC. Savedoff, W. D., R. Levine, and N. Birdsall. 2006. “When Will We Ever Learn? Improving Lives Through Impact Evaluation,” Center For Global Development, Washington, DC. Thornton, R. L., L. E. Hatt, E. M. Field, M. Islam, F. S. Dia, and M. A. Gonzalez. 2010. “Social security health insur- ance for the informal sector in Nicaragua: A randomized evaluation,” Health Economics 19, Supplement 1, September.