Costs and Benefits of Agricultural Research: The State of the Arts SWP360 World Bank Staff Working Paper No. 360 October 1979 Prepared by: G. Edward Schuh and Helio Tollini (Consultants) Agriculture and Rural Development Department Copyright © 1979 The World Bank 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. The views and interpretations in this document are those of the author and should not be attributed to the World Bank, to its affiliated organizations, or to any individual acting in their behalf. F C O P Y The views and interpretations in this document are those of the authors and should not be attributed to the World Bank, to its affiliated organizations or to any individual acting on their behalf. THE WORLD BANK Staff Working Paper No. 360 October 1979 COSTS AND BENEFITS OF AGRICULTURAL RESEARCH: THE STATE OF THE ARTS This paper reviews the issues to be faced in allocating funds between projects and between institutions involved in agricul- tural research, and between agricultural research and other activities. In doing this the authors first identify and elaborate the goals of agricultural research, including (i) increasing consumer welfare, (ii) increasing farm employment and incomes, (iii) increasing net income of the rural sector, (iv) allowing agriculture to contribute to rural economic development, (v) preserving the environment, and (vi) promoting overall rural development. The paper goes on to identify and review a wide range of methods and procedures that might be used for evaluating the con- tribution and effectiveness of agricultural research programs. These include (i) those which might be used to evaluate research efforts that have been underway for some time, and (ii) those which are appropriate in attempting to evaluate the research that is proposed but not begun. The paper then examines some of the implications of using such tools in evaluating research. Prepared by: G. Edward Schuh and Helio Tollini (Consultants) Agriculture and Rural Development Department Copyright © 1979 The World Bank 1818 H St., N.W. Washington, D.C., 20433, USA PREFACE This paper is based on a larger report prepared for the Consultative Group on International Agriculture Research (CGIAR). When this paper was written, G. Edward Schuh was Director, Center for Public Policy and Public Administration, Purdue University. Helio Tollini was Visiting Professor, Department of Agricultural Economics, Purdue University, on leave from the Brazilian Agricul- tural Research Corporation (EMBRAPA). CONTENTS Page SUMMARY I. INTRODUCTION 1 A. Why Evaluate and Analyze Agricultural Research? 1 B. Difficulties Inherent in Evaluating and Analyzing Research 3 C. Some Conundrums 5 a. Definition of Outputs 5 b. Definition of Inputs 8 c. Tangible Output of the Research Process Versus Effective Adoption at the Farm Level 9 d. Ex ante Versus Ex post Considerations 10 e. The Role of Economic Policy 11 f. Negative Results 12 g. Joint Outputs 12 11. SCOPE OF AGRICULTURAL RESEARCH 14 A. Increasing Agricultural Output 14 B. Quality Changes in Products 15 C. Conservation or Saving of Inputs 16 D. Improvements in the Marketing System 17 E. Improvements in Supply Industries 17 F. Institutions and Economic Policy 18 G. The Distribution of Income 19 III. THE ROLE AND IMPORTANCE OF GOALS 19 IV. THE SIDE EFFECTS OF TECHNICAL CHANGE 22 A. Income Distribution Consequences 22 B. Employment Effects 23 C. Environmental Consequences 24 V. METHODS AND PROCEDURES FOR EVALUATING AGRICULTURAL RESEARCH 24 A. Ex post Studies 25 a. The Inputs Saved Approach 25 b. The Use of Consumer and Producer Surplus 26 c. The Production Function Approach 33 d. Impact of Research on National Income 34 e. Nutritional Impact 35 B. Ex ante Models 36 a. Scoring Models 38 (1) The Iowa Model 38 (ii) The North Carolina Model 40 (iii) The NASULGC-USDA Model 41 b. The Minnesota Model 43 c. Pinstrup-Andersen and Franklin Model 44 d. Cartwright Model 46 e. Castro and Schuh Model 47 f. Easter and Norton Model 49 g. Atkinson-Bobis Model 50 VI. CONCLUSIONS AND RECOMMENDATIONS 53 REFERENCES 56 APPENDIX A - The Iowa Model 65 APPENDIX B - The Minnesota Model 68 SUMMARY 1. That agricultural research and the knowledge so generated is an important means of raising agricultural productivity is widely accepted. As the resources committed to such research increase, important questions of resource allocation inevitably arise since these resources h'ave com- peting uses. Data on the costs and benefits of research prov.ue the means of handling these allocation questions. 2. There are few market signals to aid decision-making in agricultural research, and cost-benefit analysis can help to make good this deficiency. It can help research managers improve the efficiency with which their re- sources are used. It can guide decisions on setting appropriate levels of budgetary support. And it can improve the relevance of research to agricultural development. Basic Problems and Issues 3. The authors note inherent difficulties in co.iducting cost-benefit studies of agricultural research. One such difficulty is that of seren- dipity or solving a problem through a chance discovery. This character- istic of research makes it difficult to know how to predict when such an insight or discovery will occur, especially since such chance discover- ies may occur in fields of inquiry rather far removed from those where they have direct application. 4. Another difficulty is that creativity may be stifled if too much emphasis is given to cost-benefit analysis since it may induce the re- searcher to direct his labors to those activities that have obvious and easily understood results. Similarly, the use of cost-benefit analysis may reduce the incentive for basic research since the output of the re- search process in this case tends to be some relatively abstract know- ledge whose relevance may not be readily apparent. 5. There are also a number of operational difficulties in making a cost-benefit analysis of agricultural research. For example, it is hard to define the output of the research process so as to be able to measure the benefit; and also to define the input of the research pro- cess for measuring costs. One has to assess the effective adoption of an innovation at the farm level instead of measuring the immediate out- put of the research process. It is necessary to decide whether the goal is to evaluate a research effort after the fact, or whether the evalua- tion is to be used as a guide to planning research on the basis of ex- pected future benefits and costs. The influence of economic policy on the adoption of the new innovation has to be isolated so as to measure the true effects of the research effort. It is necessary to judge, whether a negative research result is an addition to knowledge or a zero output. Side benefits of the research process, such as improvement of skill levels of researchers and the contribution that a strong research center can make to strengthening other research centers, are difficult to measure. - ii - 6. The authors see recognition of the various contributions that re- search can make as the key to evaluating the benefits of agricultural research. To increase agricultural output directly is the most widely recognized contribution. But research has other effects. It may improve the quality of agricultural products, conserve or save production inputs, bring about improvements in the marketing system, reduce the cost of farm inputs or improve their quality, improve institutions and economic policy so as to facilitate technical change, and influence the distribution of income. 7. To make a cost-benefit analysis of agricultural research properly the goals of the research program have to be specified. Goals can be set at various operational levels. At one extreme are rather general goals such as growth, equity, and security. AtIthe other extreme are such goals as to produce a more fertilizer-responsive variety or a more effective insecticide. Intermediate, and perhaps more pragmatic, goals include: (a) increasing consumer welfare by increasing food supplies; (b) increasing the income and employment of farm workers; (c) increasing net income of the agricultural sector; (d) increasing the contribution of agriculture to general economic development; (e) preserving the environment; and (f) expanding the income and employment opportunities of rural people. 8. The paper recognizes that all of these goals are not attainable simultaneously. Moreover, the importance of individual goals will depend on both the particular set of policies used to promote general economic development and the stage of development of the individual countries. It is for these reasons that the goals should be carefully specified. 9. The final set of issues is the side effects of technical change. Some of these side effects will be positive and thus contribute to the benefit side of agricultural research. Others will be of a negative nature, and must either be added to the cost side of the research pro- cess or subtracted from the benefit side. Among the side effects that need to be considered when increased output is the primary goal are the impact of technical change on the distribution of income, its impact on employment, and the consequences to the environment. - iii - Summary of Models and Procedures 10. This papaer identifies a wide range of methods and procedures now available for evaluating the effectiveness of agricultural research pro- grams. These methods and procedures can be divided into two groups: (a) those which attempt to evaluate research efforts that have been underway for some time, and in an after-the- fact sense; and (b) those which look to the future and attempt to evaluate the research effort before the fact. The paper discusses examples of each class in some detail. Data generated from each of the two perspectives can improve decision-making with respect to agricultural research, and can also provide the justifi- cation for budget support of research programs. 11. The various procedures used to analyze research after-the-fact generally attempt to assess the effect of the research effort on agri- cultural output or on resource savings, and combine quantitative assess- ments of these effects with costs of bringing about the observed changes. The analysis may focus on a particular innovation, such as hybrid corn research in the United States or the development of improved cotton in Brazil, or it may focus on the total research effort in a state, country, or region. It may also attempt to evaluate side effects such as resource displacement, employment generation or the creation of unemployment, and specific environmental impacts. 12. The strengths of this particular approach to the problem are that it draws on prior knowledge of how the economy functions and on empirical knowledge of the actual effect of the research on the economy. In addition, it can apply modern statistical procedures to separate the effect of the research effort and to control in a systematic way for other variables that may be affecting either the research output or agricultural produc- tion. Such procedures enable one to obtain a more precise assessment of the particular contribution of the research effort. 13. The disadvantage of these procedures is that they need a lot of in- formation, as for example on input and output data on crops and livestock and on the particular innovation being considered. In addition, if the objective is to control for other influences, such as research programs in other countries and in related research fields, data that measure these other variables are required. 14. The use of such procedures provides an important means of adding to the stock of knowledge about the effect of agricultural research on econ- omy at large. To the extent such knowledge is transferable from one sit- uation to another, it provides information useful for improved decision- making. Moreover, refinements have been made in these procedures to the point where commodity and regional specificity can be obtained in pro- viding guidance to decision-makers, and in a form that is useful for ass- essing the value of research as an investment. - iv - 15. The paper also reviews procedures which attempt to evaluate research before-the-fact, which are shown to be equally numerous and well-developed. Such procedures are widely used in non-agricultural research, particularly by the private sector and for military technology. The survey concen- trates on the procedures used for agricultural research, although they note that the use of such approaches is still at an early stage. 16. The paper sketches the procedures which attempt to evaluate research before it is undertaken, which generally start with a specification of goals and objectives for the research. An attempt is then made to assess the contribution of alternative research projects to the attainment of these goals, and to assess the costs of attaining the goals under these alternatives. The rankers or "judges" may be scientists themselves, informed lay people, or some combination of the two. The particu- lar techniques used range from a simple scoring of individual re- search projects to complicated formal mathematical models. 17. An important subset of the before-the-fact procedures draws on other knowledge of the economy, just as do the after-the-fact pro- cedures, but poses the analytical question in a forward-looking rather than a backward-looking perspective. Procedures are now available which permit such forward-looking analyses without the use of full or complete systems. Alternatively, simulation models can be developed which per- mit the rapid evaluation of a large number of alternative research pro- jects, and in such a way as to analyze expected detailed effects on the economy. 18. As is pointed out, one strength of these procedures is that they look to the future and hence provide a more realistic and relevant guide for decision-making. They also provide a mechanism for systematically considering goals and objectives of the research program, and with considerable flexibility. And they provide a rigorous and dis- ciplined means of pooling the judgment of large numbers of informed people. 19. One disadvantage of the before-the-fact procedures is that they are costly in terms of the input of time required of informed professionals. They are both time-consuming and complicated, which may explain why the more complicated procedures for the most part have been used only once. The utility of the procedures is also heavily dependent on future devel- o0mqnts actually turning out as expected. Moreover, the informed judgment which is expected at the end of the analysis can be no better than the quality of the input which goes in. If nothing more is accomplished thafi to pool ignorance, little of value will be provided as a guide to improved decision-making. 20. To conclude, it is confirmed that, although an impressive array of procedures for evaluating research have been developed, none of them offers a panacea, nor could they be used in a mechanical way as a guide to decision-making. However, the development of alternative pro- cedures appears to have far outpaced their actual use for decision-making, especially in publicly-supported agricultural research. I. INT CTION 1.1 This paper is devoted primarily to a diacussion of how the decision-making process with Jiepect to resource allocation decisions in agricultural research sight *,lAproved. The discussion deliberately eschews the rather narrow concept. of cost-effectiveness, with its conno- tation of "are you being productive?", in favor of a more general review of the problem of evaluating agrica,ltural research and of establishing research priorities. 1.2 Allen (5) makes a useful distinction between resoarch manage- ment and reseatch administration. Besearch management involves essentially three sets of decisions: (1) the awount of funds to be allocated to re- search; (2) the choice of research problems to be investigated with available funds; and (3) the appropriate research strategy to be employed in the in- vestigation of a given problem. Research administration, on the other hand, involves the day-to-day decisions required to efficiently carry out the research task. 1.3 Cost-effectiveness is associated with research administration, and involves questions of the mix of assistants versus professionals, choice of instruments and equipment, and so forth. We stay away from such questions, in large part because so little research has been done on how to efficiently organize agricultural research. The cost-benefit framework which we have chosen, and the procedures based on it that have been devised for ex ante decision-making, are ouitable for dealing with theo anagesent question which in our view is the key issue. 1/ This paper reviews the various procedures available under theme two rubrics, 1.4 This section attempts to do three things: (1) explain why an evaluation and analysis of agricultural research may be useful; (2) dis- cuss the diffictlties inherent in making such an evaluation; and (3) examine some of the conundrums one faces in operationally doing such evaluation and analysis. A. Why Evaluate and Analyze Agricultural Research? 1.5 The notion that agricultural research and the knowledge so gen- erated is an important means of raising agricultural productivity is now widely accepted. International development agencies curre tly give agricul- tural research a higher priority than they did some ten ye4rs ago, and devel- oping countries are making a stronger commitment to strengqhening their agricultural research capability than they have in the past. 1.6 World expenditures on agricultural research are now on the order of approximately $5-6 billion per year. At the resource commitment to agri- cultural research becomes progressively larger, important questions of re- source allocation inevitably arise. Resources Used for this purpose have alternative uses, and if developmental resources are to make their maximum contribution to improving the welfare of an ever growing world population, 1/ Lindner (56) has an excellent discussion of the issues involved between research management and research administration. they have to be used efficiently. To put it somewhat differently, research is an economic activity requiring the organization of scarce resources. 1.7 An analysis of costs and benefits provides the means of handling these allocation questions. We choose to frame the discussion in terms of these concepts rather than the more narrow concept of cost effective- ness since it enables us to discuss a wider range of problems. Moreover, the concepts of costs and benefits can be extended to include or incorporate (at least in principle) the various qualitative dimensions of agricultural research. 1.8 The need for a cost-benefit analysis of expenditures on agricul- tural research arises logically as a consequence of the lack of efficient markets to allocate resources for this purpose. There are no readily avail- able markets for the output of a research process. Prices are not established for the product of the research activity, in part because much of such knowl- edge is free to the user once it becomes available. 1.9 This lack of an efficient market has two important consequences. First, there are little by way of market signals to allocate scarce research resources among alternative uses. Consequently, such alloca- tions tend to be made largely by administrative decree. Second, in the absence of a benefit-cost analysis, society tends to under-invest in agricultural research in the aggregate. Since many of the benefits of agricultural research cannot be captured by private firms, they will allocate fewer resources to this use than they otherwise might. Equally as important, government agencies in allocating resources purely on the basis of "informed" judgment may also allocate insufficient resources to this endeavor. In fact, the observed high social rates of return to past investments in agricultural research suggest just such an under-investment. 1.10 The allocation question is important, for within the possible range of agricultural research activities, some activities will have a higher pay-off than others. To maximize social output, resources should be allocated where their contribution is greatest at the margin. 1.11 The investment question is important because an under-investment in agricultural research will mean the sacrifice of potential output and the possible improvement in human welfare. Similarly, over-investment in research, should this occur, has comparable consequences, except that the sacrificed output and human welfare arises because of the failure to use the resources in alternative uses. 1.12 Cost-benefit analysis provides a means of generating decision information that would normally be generated by a market or markets. Hence, its contribution is to assist in obtaining a more efficient allocation of resources within the research sector, and to assist in obtaining the optimum allocation of resources to agricultural research in the aggregate. - 3 - 1.13 At the most elementary level this issue is one of priorities. Some, of course, would argue that no attempt should be made to establish priorities in research, for in their view the task is too difficult. But that argument is largely academic. Priorities are established all the time, if by no other means than that resources are not available to do all that one might want to do. The fundamental question is whether systematic analysis by means that lead to quantifi- able decision variables can improve the decision-making that inevitably takes place. 1.14 There are other reasons than the allocatory and investment questions for desiring to evaluate and analyze agricultural research. For example, the diffusion of one technical innovation may increase the potential pay-off from another innovation. If the research process is being monitored and evaluated, these potential high pay-off opportu- nities will be more quickly identified. 1.15 Second, as will be noted below, technical change often has some deleterious consequences associated with it in the form of unexpected shifts in the distribution of income, the premature release of labor from agriculture, and effects on the environment. If these consequences are identified at an early date, corrective measures can be adopted. This will facilitate the adjustment to the new situation, thereby making for a higher net social payoff, and also avoid unnecessary and often ill-informed criticisms of the technology itself. 1.16 Finally, changes in economic policy and modifications of the environment are often required to make the diffusion of a given tech- nical innovation possible. Monitoring and assessing the research process and its output will identify these changes and modifications and provide the basis for bringing them about. These, in turn, will make agricul- tural research a more effective contributor to agricultural development. 1.17 To conclude, there are three important benefits from monitoring and assessing the research process: (1) it provides a basis for justify- ing budget support, thereby leading to investments in the appropriate amounts in agricultural research; (2) it provides the basis for making a more efficient use of the resources allocated to agricultural research; and (3) it provides the basis for making agricultural research a more effective contributor to agricultural development. The state of knowledge on agricultural research has now progressed to the point where many of these benefits can be realized, if resources are allocated to the appro- priate analytical and empirical research. B. Difficulties Inherent in Evaluating and Analyzing Research - 4 - 1.18 Recognition of the possible contributions from assessing and monitoring agricultural research does not clear the path to easy empirical work, nor make the decision-making process any easier. There are inherent difficulties in making cost-benefit studies of agricultural research, and at least three of them deserve special mention at this point. (Other difficulties will be noted in the next section.) 1.19 The first inherent difficulty is that of serendipity. Research is not a mechanical process that can be programmed as an ordinary indus- trial process. The very fact that a problem requires research implies that there are some unknowns in the system. A solution to a problem is often found through luck or a chance discovery. Such discoveries often make possible quantum jumps in science. 1.20 This problenm gives rise to difficulties in attempting to assess the cost-benefit ratios or cost effectiveness of research, especially when the goal is to make ex ante analyses (in contrast to ex post) as a means of guiding research decisions. The difficulty is in knowing how to predict when such an insight or discovery will occur. The problem is doubly difficult in that such chance discoveries may often come from activities in fields rather far removed from where the dis- covery has direct application. 1.21 Serendipity is related to creativity, of course, and this brings up another issue. There are inherent difficulties in attempting to program research, since excessive programming can stifle creativit:'0 I/ The need to justify in economic terms every endeavor he undertakes can cause the researcher to direct his labors to those activities that have obvious and easily understood benefits, while neglecting those that have more risk. The inhibition of entrepreneurship that results can have important long-run implications. 1.22 Finally, there is the inherent difficulty with basic research. Benefits from applied research can be rather easily specified, oince such research typically has rather well-defined objectives that are expressed in terms that can for the most part be readily measuredo The situation with basic research is not so felicitous. The output of the research process is generally some relatively abstract knowledge whose relevance may not be readily apparent. Moreover, by definition such knowledge may have applicability over a wide range of problem- solving research. 1.23 Making ex ante judgments of the value of such knowledge is quite difficult, and there are limitations in assessing it even in an ex post sense. Yet such research may ultimately have the highest payoff to society. Potential growth would be sacrificed if such research were not undertaken because of. the difficulties of assessing either the benefit or cost stream. 1/ For a cogent discussion of this problem, see Schultz (86). -5- C. Some Conundrums 1.24 Difficulties discussed in this section are of a more opera- tional nature. They involve both problems of concept - how one defines what one is after - and problems of measurement. Each presents diffi- culties in making cost-benefit analysis an operational tool for guiding the allocation of agricultural research resources. a. Definition of output 1.25 In principle the research process can be viewed just as any other production process. Inputs of various kinds are combined in rather parti- cular ways to produce an output. The problem is in knowing how to define the output. 1.26 At the most abstract level, most would probably agree that the important output of the research process is new knowledge. If there were a market in which this new knowledge were bought and sold, and the output were in identifiable units, the measurement problem would be relatively straight-forward. One could measure the number of units produced and multiply it by the price determined in the market, and a measure of the total value of output would be at hand. 1.27 Unfortunately, the world is not that simple. Knowledge is intangible and in some respects undefinable, although in many respects we may recognize it when we "see" it. But reaching agreement on what a unit of knowledge is would be quite difficult. 1.28 Similarly, no well-defined market exists for knowledge, despite the fact that knowledge obviously has economic value, and that individuals who produce it tend to be rewarded in proportion to their ability to produce it. Hence, at this rather abstract level we are left in a situa- tion in which it is difficult to agree on the unit we want to measure, and there is no mechanism which gives us a direct measure of the value of the unit to society. 1.29 Because of these difficulties, researchers attempting to evaluate research have used a number of different surrogates for the output of the research process, each of which has certain strengths and weaknesses. One approach is to recognize that the knowledge produced by the research process is typically published in the form of scientific and technical papers. The number of such papers then serves as a proxy for the output variable. 1/ 1.30 An advantage of this approach is that it does provide a number to work with. Moreover, to the extent the publications are screened by knowledgeable scientists as a basis for publication, there is some assurance that a publication reflects something called "new knowledge". 1/ See Chapter 5 of Evenson and Kislev. (34) - 6 - 1.31 By the same token, however, it is clear that the "amount" of knowledge transmitted in a published paper varies a great deal from one paper to another. There are serious difficulties in attempting to place relative values on the publications, and the most that has been done to date is to assume that "a publication is equal to a publication". 1.32 It should also be noted that the review process for publication io itself imperfect. Well-intentioned researchers can disagree over whether a given publication contains information that advances our knowledge or not. We are all aware of the unorthodox but important idea that is rejected for publication. And similarly, we are all aware of the paper that gets published because of the name attached to it, rather than the content it carries. 1.33 The use of number of publications as a measure of the research output can also have pernicious long-run consequences on the research industry, even though it may have value in an ex post check of what has transpired. The problem is that researchers can easily play the "publi- cation game" if numbers of publications are used as an index of output and productivity. Publications can be broken down into smaller units, different perspectives on the same project can be published in separate journals, and so on. 1e34 An alternative approach is to define the output in terms of some well-specified innovation. This moves the concept of output closer to something that has more immediate economic value, and it provides an index about which it might be easier to reach agreement. Moreover, an innovation can be related more directly to the original objective or objectives of the research process. Examples of such innovations include hybrid corn, an improved inbred line, an insecticide that treats a particular pest, a fungicide that controls a disease, etc. 135 The difficulty with this concept, of course, is the problem of non-adoption. In some sense an innovation has to meet a market test, If the innovation is not adopted because it is too expensive, as for example in the case of a pesticide, the research has not attained its full objectives, even though in terms of a particular research project it may have attained its goal. In this sense, to measure the output of the research process in terms of innovations generated can be misleading, or at best not tell all of the story. 1.36 For this reason, most attempts at evaluating the output of a research process have concentrated on measuring that output in terms of its impact on the production process. As will be noted below, this involves making some independent estimate of the extent to which the research has shifted the production function or the supply function for the particular crop or livestock category. The analysis is formulated in terms of a particular innovation, but the empirical work is in terms of resources saved, shifts in the production function, or shifts in the supply function, The economic calculus can then be applied to estimate the flow of benefits. 1.37 A particular example of this approach is the evaluation of tnu social rate of return to agricultural research by means of the Hayami- Ruttan metaproduction function. 1/ This production function includes social or public inputs in addition to those used directly on the farm. As a measure of the research input, the number of scientists working at agricultural research, or at research and extension, is introduced as a separate variable in the function. 2/ By statistical means, then, the contribution to agricultural output of this particular set of inputs can be isolated, and an estimate of either the productivity of the research inputs or the social rate of return to investments in research can be made. Similar to the case immediately above, the output of the research effort is measured in terms of its effect on agricultural output. 1.38 Despite the complexity which has emerged from the above discus- sion, only the simple case has been treated - the case in which the principal contribution of the research is expected to be an increase in agricultural output. More complex outputs of agricultural research include qualitative changes in the product, such as a tomato that transports better, or an improvement in the nutritional value of a product. 1.39 Similarly, very little progress has been made to date in evaluat- ing the returns to economic research. Here the gamut is quite wide: farm management research which leads to a more rapid adoption of technical innovations, marketing research which improves price incentives to farmers, research which leads"to improved understanding of economic behavior, and policy research that leads to an increase in agricultural output from a given bundle of inputs due to the removal of policy distortions. Research to devise a suitable methodology for dealing with this class of problems has not made much progress, largely because of the difficulties encountered in defining the output of the research process. 1.40 Finally, there is the problem of maintenance research. Certain agricultural innovations are subject to a relatively high rate of obsolescence. A new wheat variety becomes susceptible to a new strain of rust. Insects de- velop resistance to insecticides. And antibiotics lose their effectiveness in combatting a particular disease. 1.41 The consequence of this obsolescence is that a certain amount of ongoing research is required to just "stay even," in contrast to advancing the frontiers of knowledge or even the efficiency frontier. The output of such maintenance research is just as important as research which pushes the knowledge frontier further out. 3/ Identifying it and taking it into account may be somewhat more difficult, however. 1/ For an example see Thompson (98). 2/ Alternative variables such as total expenditures on research can also be used. 3/ Ayer and Schuh (10) argue that the high payoff to the cotton research program in Brazil was due in part to the speed with which it developed varieties that were resistant to a devastating infestation of wilt. -8- b. Definition of inputs 1.42 For certain categories of inputs the definitional and measurement problems on the input side are more straight-forward. For example, the research process typically involves the use of certain labor inputs, including skilled professionals, physical inputs such as land, buildings, and capital inputs, and agricultural inputs such as research animals and fertilizer. With appropriate accounting procedures these inputs can be measured with a fair degree of accuracy, although difficulties arise when it is necessary to attribute fractions of the services of skilled labor and capital equipment to particular research endeavors. 1.43 The difficulties arise in knowing how to conceptualize and measure the inputs of certain skilled manpower, and in knowing how to treat the existing stock of knowledge. In treating skilled manpower, two problems arise. If labor markets were efficient and without distortion, the salary paid the scientist would be a suitable measure of the value of his services. However, labor markets for skilled manpower are neither efficient nor free of distortions. Hence, in some cases a more suitable approach would be to attempt to shadow price the labor input. V 1.44 The second problem is to know how to measure the chance idea that comes from an encounter with another researcher. To the extent that this is a free good, there is no problem. It enters as a free good. To the extent a consulting service is involved, the question becomes more compli- cated if the labor market is not efficient, These problems in pricing the services of scientific manpower are for the most part at least tractable. The more difficult problem is to know how to price and measure previous research, The very successful cotton research program in Brazil had its start with lines brought from the U.S. 2/ To what extent should the cost of producing those lines be considered in evaluating the benefit-cost ratio in Brazil? The procedure followed by Ayer and Schuh was to treat the lines as a free good. 1.45 Similarly, there was considerably research in the UoS. and Japan that was drawn on in producing the Mexican wheatso How far back should the analyst go in imputing the costs of these programs to the Mexican wheat program? If the outputs of these previous research endeavors are treated as a free good, the social rate of return to the Hexican wheat program would be exceedingly high. If all the costs leading up to that particular innova- tion were included, the return would be substantially less, 1.46 When the research endeavor is well-defined and institutionally specific, as in the case of the Mexican wheats, operationally the answer seems relatively straight-forwardo But if the Mexican wheat$ had by chance been produced as a logical consequence of either the U.S. or Japanese endeavors, the answer would not have been so clear. This brings out the 1/ Shadow prices are measured either implicitly, or in terms of the oppor- tunity cost of the input. They are used when observed merket prices do not reflect the true scarcity value of the resource to the economy. 2/ See Ayer and Schuh (10). - 9 - difficulty in knowing how to treat the important input of past knowled!ge. H 1.47 A similar although somewhat different problem arises when one considers a research organization like the land grant colleges. Typi- cally, a considerable amount of basic research will be going on in one part of the university, while the school of agriculture will be con- centrating more heavily on the applied side. The synergism among researchers can produce a substantial amount of knowledge that is "free" to the applied research program. The question is how to take account of this when assessing benefit-cost ratios for the research program. If only the inputs involved in the applied program are considered, the results will be of one dimension. If all research at the university is included, the answer will be quite another. To date there are no easy answers to this important imputation problem. 1.48 Evenson, Flores, and Hayami (33) have handled this "transfer" problem by regressing the change in yield on a series of knowledge stock variables that represent research activities in related disciplinary and commodity programs and research activities in other geographic areas that might be transferable. Hence, statistical procedures are used to isolate the separate effects of the various programs, and the costs and benefits can be computed accordingly. 1.49 A final problem with inputs arises outside the research process per se. The adoption of a new innovation can induce shifts in resources among sectors. If one is using secondary data to estimate the benefits of the research program, some attempt has to be made to take account of these resource sLifts. A difficulty arises because labor is often an important component oi the resource shift, and labor markets often are imperfect. Conse- quently, the observed wage rate (or rates) may not be an accurate measure of the tLue opportunity or social costs of the labor. Some method of shadow pricing then has to be used. c. Tangible Output of the Reseac'. Ptortss Versus Effective Adoption at the Farm level. 1.50 This problem was dlscussed bil -ly above. But a somewhat more systemaLic treatmenit might be helpful. Oine way to assess the cost- effect,veitess of a research program is In terms of the objectives of the re-Žirch effort. These could be specified in quite precise and easily understood terms: (1) to produce a variety or varieties of a crop that has double the yield potenrial of existing varieties, (2) to find a means of controlling a particular insect, or (3) to find a means of tontrolling a particular disease. A management-by-objectives approach to the assessment and monitoring problem would then assess whatever cost it took to attain the specific objective. 1_/ It should also be noted in thiis context that the output of the Brazilian and Mexican programs cited are logically outputs of their predecessor programs. Thie conceptual and measurement problem in some sense, then, is a problem of identifying and measuring the value added from the respective programs. - 10 - 1.51 But the mere attainment of the objective - which typically can be expressed in terms of an innovation at the farm level - does not guarantee that attaining that objective is relevant to the farmer. For example, the crop variety with double the yield potential may not fit into existing cropping patterns, or as with the early IRRI rice varieties, the new plant may not have suitable consumer qualities. In the case of the high lysine gene in corn, the research objective of introducing the gene was attained. But high lysine corn is still not a commercially viable enterprise because yield objectives have not yet been attained and there are problems of product identification or differentiation in the market. 1.52 Similarly, the system for controlling an insect may be too costly in terms of chemical and labor inputs, or it may have undesirable environ- mental consequences. The same applies to new means for controlling a disease. 1.53 One solution to this problem, of course, is to place the appro- priate specificity on the research objectives. Hence, the goal would be to attain economically viable varieties and systems of insect and disease control. Similarly, the appropriate environmental constraints can be placed on the research objectives. Although feasible, the general use of such specificity would for the most part be a departure from present practice. 1.54 In the final analysis, innovations have to meet the market test when an attempt is made to evaluate the cost effectiveness of a research activity. Equally as important, they have to be consistent with whatever institutional restrictions the body politic has put on innovative activities. These considerations suggest that the cost-benefit or cost effectiveness analysis has to be made in economic terms and with institutional constraints, in contrast to the more common operational objectives of a research project. d. Ex ante Versus Ex post Considerations 1.55 Benefit-cost analyses of agricultural research can be made either in an ex post sense or in an ex ante sense. To date, most of them have been made from an ex post perspective, since their primary objective has been to assess the role of agricultural research in economic development, and to determine whether investments in such activities have been economically viable. We are aware of four ex ante analyses of agricultural research, 1/ although as will be noted below there is a stronger tradition of ex ante analyses for the industrial sector. 1.56 Which of these is desired depends on the purpose of the assessment or monitoring. If the goal is to determine how efficient particular research institutions are, or how efficient particular lines of research activity have been, then an ex post approach is required. If the goal is to use benefit-cost analysis as a guide to how research resources should be allo- cated to maximize their payoff, an ex ante perspective is required. 1/ Castro and Schuh (75), Easter and Norton (29), Klein (53), and Pinstrup-Andersen and Franklin (71). - 11 - 1.57 Although not used on an extensive scale for this purpose, we believe substantial emphasis should be given to ex ante analyses. After all, an important question is to know how many resources could be allocated to attain a given research objective. Moreover, one would like to know before the fact where the expected payoffs are likely to be the greatest. This, of course, is not to deny the value of ex post analyses, for they can provide important insights into the research process, and provide a basis for compar- ing alternative organizational and methodological approaches. 1.58 Somewhat different methodologies are required to make an ex ante analysis than is required for an ex post analysis. Moreover, the precise- ness and robustness with which inferences can be drawn are quite different in the two cases. We will return to these problems below. e. The Role of Economic Policy 1.59 The role of economic policy has not been given a great deal of attention in past benefit-cost analyses of agricultural research. Ayer and Schuh (10) and Schuh (83) have noted the role of economic policy in determin- ing the distribution of benefits from research between consumers and pro- ducers in the society. But only Hertford (49) and his colleagues have to our knowledge systematically addressed the question of whether economic policy affected the rate of return or the benefit-cost ratio of research. 1.60 Clearly, economic policy can affect the measured benefit-cost ratio of research, and in a number of important ways. First, to the extent that economic policy distorts price relatives it may cause a failure to adopt innovations that would otherwise be adopted. An important example occurred in Brazil when trade and other policies lowered the domestic price of agricul- tural products relative to the price of fertilizer. Since under the circum- stances the use of fertilizer was not profitable for many crops, a consider- able amount of soil and fertilizer research went for nought. 1.61 The consequences did not stop there, however. Since the use of fertilizer on corn was not profitable under the prevailing price relatives, farmers also failed to adopt hybrid corn. Hence, the return to what was a relatively effective research program on hybrid corn was quite low. 1.62 Clearly, economic policy caused Brazil to undervalue its research activities. What might have been a relatively viable research effort was perceived as a low payoff activity. Hence, Brazil invested less in agri- cultural research than it might have under a different policy regime, and sacrificed this important source of growth. 1.63 The role of economic policy in influencing the observed pro- ductivity or return from agricultural research can perhaps be seen more directly in a somewhat different context. If the goal of a research program is to increase the output of a particular crop or crops, the benefit-cost analysis or evaluation of cost effectiveness may require placing a value on the increase in output. If economic policy has distorted relative price ratios, the contribution of the research program can be either under- or - 12 - over-estimated, depending on the direction of the distortion. 1/ To eva- luate the benefit-cost ratios in this case some estimate of shadow prices is required. 1.64 Economic policy has still other effects on the perceived or actual benefit ratios of agricultural research. For example, policy measures which restrict exports of agricultural exports can reduce the earnings poten- tial of the research in terms of exchange earnings, thereby causing the actual benefits from the research program to be substantially reduced. The reduc- tion in foreign markets can reduce the income and employment generating potential of particular innovations. And trade policy can influence whether it is low-income consumer groups that receive the benefits of research or upper-income producer groups. Each of these are factors that should be taken into consideration in assessing the productivity and cost-effectiveness of research. Clearly, the particular economic policy regime has a major impact on the potential, perceived, and actual returns from agricultural research. f. Negative Results 1.65 The effectiveness of agricultural research is typically evaluated in terms of its successful contribution to specified research goals and objectives. An important conundrum is how to handle the problem of negative results. Several years of concentrated effort may have failed to increase the yield of a particular crop. But the research effort itself may have clearly shown that a number of possible approaches to raising yields is not viable. The problem is to know how to handle such negative results. 1.66 Clearly, there has been an increase in knowledge, for researchers now know at least part of what will not work. This is of value to present and future researchers. But in terms of attaining the operational goal of the research, it has not made a contribution. Ultimately, of course, this problem comes down to the concept of output used to measure the effective- ness of the research program. In terms of additions to the stock of know- ledge, the research effort may have been quite productive. In terms of an on-the-farm gain in agricultural output, little may have been accomplished. g. Joint Outputs 1.67 The final conundrum has to'do with the fact that a research pro- gram inevitably involves joint outputs. For example, researchers acquire new skills and knowledge by the very process of engaging in research. This increase in human capital has considerable value to society and should be taken into account in assessing the productivity of the research effort. 1/ India has substantially over-valued wheat in the domestic economy, compared to international opportunity costs, while substantially under- valuing rice. This has undoubtedly affected the perceived rates of return to its respective research programs, and probably has affected the relative rates at which new innovations have been adopted. See Sukhatme (95). - 13 - 1.68 Similarly, research programs typically have a training or educa- tional program associated with them. The complementarity between teaching and research is well recognized, and the teaching is assumed to be better if it is associated with a vital research program. Some part of the educa- tional output should therefore be attributed to the research program. 1.69 Similarly, there are important institutional spillovers from an effective research program. The contribution of the International Centers in strengthening national research programs and systems is an important example. The knowledge, skills, and experience imbedded in the researchers at the Centers presumably have value to the national programs. Similarly, a re- searcher may take on special value as a consultant to government or other researchers as a result of his participation in an effective research pro- gram. These contributions can be an important output of a research program, and should be taken into account in assessing its productivity. To date, however, such considerations have not been included when evaluating the effectiveness of research programs. - 14 - II. SCOPE OF AGRICULTURAL RESEARCH 2.1 Priorities for support to agricultural research have been reason- ably well defined. The emphasis at the present time is primarily on in- creasing the output of food, and within that larger mission to improve the production technology for a specific list of crops and'anumal categories. Other considerations are deemed relevant, however, such as increasing farm income and employment, saving or earning foreign exchange, and changing the nutrient content of conventional food crops. 2.2 It is not the intent of this section to take issue with these priorities or objectives. However, a rather broad specification of the scope or range of activities through which agricultural research can operate will help set the framework for later discussion. This broader specification is relevant even if the benefit-cost ratios or cost effectiveness of research are evaluated on a narrower base, since some of these other dimensions may be attained incidentally to the primary mission. To the extent they are, they should be considered in the evaluation of the research program. 2.3 The broader perspective is also important in a somewhat different context. Ultimately, allocative decisions with research or any other re- sources involve a choice among alternatives. In that sense it is important to remind ourselves that research resources might be allocated to activities other than those presently agreed upon as a means of attaining the specified goals, and that these other'alternatives provide an important basis of com- parison. 2.4 The following paragraphs provide a brief discussion of the range of activities to which research resources might be allocated. The specifica- tion is not exhaustive, although it does attempt to be reasonably comprehen- sive. A. Increasing Agricultural Output 2.5 This is probably the most widely accepted operational goal for agri- cultural research. It is perhaps best typified by the motto that appeared in the legislation establishing the U.S. agricultural research system: "To make two blades of grass grow where one grew before". 2.6 But agricultural research can be used to increase agricultural out- put in a number of different dimensions. 1/ Although most national programs stress the importance of increased output of food crops and commodities, research can just as well be directed towards increasing 1/ In addition to the various dimensions in which agricultural output can be expanded, there are various means by which it can be expanded. For example, one approach is by raising yield potential. Another is to shorten the length of the growing season so that multiple cropping can be used. Other means are discussed below. - 15 - the output of export commodities or the output or products used as raw mate- rials for the industrial sector. For an individual country, the availability of food for the domestic consumer might be increased by increasing the exports of cotton or some other export product. In fact, this may well be the most efficient use of the world's agricultural resources, even though it may not necessarily increase the supply of food for all the world's population. 1/ 2,7 Directing the research process towards increasing the output of products that serve as industrial raw materials may have comparable effects. For example, an increase in the output of cotton, or better - an improvement in the comparative advantage of the country in the production of cotton - may enable it to increase the production of textiles. The increased output of textiles would be a means of earning scarce foreign exchange while at the same time creating additional employment for the domestic economy. 2,8 To conclude, agricultural research resources may be directed pri- marily to obtaining an increase in agricultural output. Even under this rubric, however, it may be used to increase the output of food products, of export products, or of products that serve as raw materials for the industrial sector. An increase in the output of a given crop may, of course, serve all three purposes. If it does the research process should receive credit on all three counts. 2.9 Individual countries may have preferences with respect to how they attain their particular goals. Moreover, research investments in non-food commodities may have a higher social rate of return than research investments in food commodities. B. Quality Changes in Products 2.10 In contrast to the above, the research resources may be directed to obtaining changes in the quality of a product. The most obvious example, of course, is when a breeding program is designed to improve the transport- ability of a product such as tomatoes. In effect, the output of the product available to consumers is increased by this means, but the immediate objective is rather different. 2.11 Similarly, the research effort may be directed to an improvement in the transportability of a commodity through time by increasing its storability. Again, the consequence may be to increase the total supply of the commodity available to consumers by reducing wastage and spoilage. But the immediate research objective is rather different. 1/ Research on non-food crops may, of course, release resources for an in- crease in food output. If resource productivity for non-food crops were raised, it might require less land and labor to produce these crops, and these resources would then be released for production of food crops. - 16 - 2.12 Finally, the research effort may be directed to increasing the supply of nutrients, independently of whether the total supply of food in some physical sense is increased or not. The development of high lysine corn brought this possibility to the fore. But the potential is much broader than this, although generally neglected in the discussion of research prioritieso In general it is assumed that an increase in food output will reduce malnutri- tion in the population at large, although the evidence to support that view is sorely lacking. 1/ 2413 In any case, the thrust of the research program can be towards changing or improving the quality of the products, and not necessarily just to increasing the physical output of food. C. Conservation or Saving of Inputs 2,14 The Hayami-Ruttan model of agricultural development brings this particular research thrust to the fore, since in their view the contribution of agricultural technology to facilitating the growth in output is to ease the constraint to output expansion implied by the resource that is becoming relatively more scarce as output expands. 2/ Their two immediate examples are biological and chemical innovations that ease the constraint implied by scarce land, and mechanical innovations that ease the constraint implied by a situa- tion of relatively scarce labor. 2,15 The problem is more general than this, however. The energy crisis of recent years has focused attention of the possibility of a shortage of nitrogen fertilizers, Agricultural research can deal with this possible shortage in a number of different ways. It may attempt to develop varieties of crops that are more responsive to fertilizer application. It may attempt to develop techniques whereby a larger percentage of the nutrients in the soil are made available to the plant. Or it may attempt to broaden the range of species that fix nitrogen from the air, as with the current research on graminae. 2416 In a similar way, the research program may attempt to economize on the use of water. The techniques of water control and management may be improved. Plants may be developed that are more resistant to drought; or that make more effective use of available water. And alternative cropping or enterprise patterns may be devised. 1/ For an evaluation of the impact of an increase in food supply on human nutrition, see Pinstrup-Andersen, Londono, and Hoover (72). These authors appeal for more attention to human nutrition in establishing research goals, although their own empirical findings suggest that malnutrition is primarily a problem of absolute poverty and that a larger food supply does little to alleviate the problem among low income groups. 2/ Hayami and Ruttan (47). - 17 - 2.17 Any research which results in a resource saving can ultimately lead to an increase in agricultural output. However, there is a difference in the research undertaken depending on whether the immediate objective is to increase agricultural output, or whether it is to "save" or make more efficient use of a resource. The consequences in terms of adjustment prob- lems, product mix, and efficiency growth are also quite different. D. Improvements in the Marketing System 2.18 The bulk of the emphasis in most agricultural research programs is generally placed on the production side. However, agricultural output is not food until it has passed through a system of processing and transportation to arrive on the consumer's plate. Substantial resources are used in trans- forming raw agricultural output to food for consumption in households. Moreover, some of these resources can be freed up for the production of food if improved efficiency is obtained in the food distribution system. Similarly, a reduction of waste and spoilage in the distribution system makes a larger supply of food available to consumers from a given supply of agricultural output. 2,39 The range of research activities here is quite great. Improved technologies for processing agricultural products may be developed, techniques which reduce spoilage may be developed, techniques that utilize by-products may be devised, the transportation process may be made more efficient, and so forth. The important point is that a productive research thrust in this dimension may do as much to increase the supply of food available to con- sumers as a direct increase in agricultural output. Estimates of 'the losses in harvesting and through the distribution system suggest that savings in this sector may be rather large. E. Improvements in Supply Industries 2,20 It is not sufficiently well recognized that much of the tech- nological change that is pertinent for agriculture does not take place directly in agriculture, but rather in the industries that supply inputs to agriculture. Perhaps the most notable example is in the fertilizer industry, where successive waves of technological innovation, together with a more competitive market structure, have led to dramatic declines in the relative price of fertilizers. This decline in relative price has led to a widespread adoption of fertilizer and contributed to a substantial increase in agricul- tural output. 2,21 Similar examples abound in the industries that supply mechanical power and equipment and machinery to the agricultural sector. Technical change in these industries resulted in more efficient machinery and equipment, the improved "packaging" of power units into more flexible, mobile units, and the development of completely new machines and equipment. 2,22 In the fertilizer industries, new forms of fertilizer have been developed. In the pesticide industries, new forms of pesticides have been developed. And the list could again be easily extended. - 18 - 2.23 Not to be forgotten is the substantial improvement in the quality of the human agent in most countries of the world as improved health care has been generalized through the society, education has become more generally available, and training programs and other sources of information have become available. The important issue here, of course, is that there are important complementarities between the skill levels of the human agent and the new knowledge generated by the agricultural research enterprise. 1/ Individuals have to process the new technical knowledge and adapt it to their own parti- cular situation. Their ability to do this improves as their level of educa- tion (broadly construed) improves. Moreover, important interactions between agricultural entrepreneurs and agricultural research stations can generate new knowledge in its own right. 2q24 The important point is that research and technical change in the supply industries for agriculture can be as important as research and technical change directly in agriculture in increasing the supply of food for the world. F. Institutions and Economic Policy 2,25 Social, political, and economic institutions are the means whereby inuividuals in a society relate to each other. Such institutions can dull the incentives for innovative activities, or they can sharpen them. Similarly, they can make for a society that is at peace with itself, or they can cause tensions, disruptions, and diversions of energies to fruitless endeavors. Research that leads to improved institutions can lead to increases in agri- cultural output and to higher levels of innovative activities. 2,26 For its part, economic policy is pervasive in society. It, too, influences how individuals relate to each other, determines how rewards are distributed among members of society, and in turn the incentives to produce and innovate. In the final analysis, economic policy is an important deter- minant of the level of saving in a society and in turn the level of invest- ment - the key to the rate of output growth. 2,27 As noted earlier, economic policy can largely determine whether new production innovations are adopted at the farm level, or whether they are left at the experiment station door. In effect, it can provide the means whereby the biological and physical research program will be quite productive for the society, or it can cancel it out almost in its entirety. 2,28 Research that leads to improved institutions and economic policies can contribute as much to increasing agricultural output under some circum- stances as can the more conventional biological and physical research. Perhaps the more important point, however, is the very real complementarity between biological and physical research, on the one hand, and socio- economic research, on the other. Without both kinds of research, neither will contribute very much. 1/ See Welch, Finis (104). - 19 - C. The Distribution of Income 2.29 Agricultural research, even of the biological and physical kind, can influence the distribution of income within a society. In perhaps its simplest form, it does this when it lowers the real price of food in a society. Low-income groups benefit relatively more than high income groups from such developments, and consequently the personal distribution of income is improved. 2.30 By influencing the ease with which one input can be substituted for another, research can cause a shift in the distribution of income among productive inputs as factor supplies change. By changing the relative pro- ductivity of particular input categories, it can also influence the relative share of income that goes to land, labor, and capital. 2.31 Biological and physical research are probably not efficient means of influencing the distribution of income. But it is almost inevitable that agricultural research will have income distribution consequences. Even if the main thrust of the research is not to change the distribution of income, the distributive consequences of agricultural research have to be taken into account in evaluating it. (See below.) III. THE ROLE AND IMPORTANCE OF GOALS 3.1 Any attempt at assessing or monitoring the cost effectiveness of agricultural research has to address the question of goals. One cannot say how effective a research program is without at the same time making an attempt to state what the goals of the effort were. If the question is turned around and the benefit-cost analysis used to establish priorities for the research effort, the same comment applies. One cannot establish priorities without saying what goals the research is supposed to attain. '3.2 In the previous section a discussion was presented of the various directions that agricultural research could take. In the present section the focus is on the goals of agricultural research. 3.3 These goals can be set at various levels of abstraction or opera- tionality. As will be noted below, Iowa State University takes growth, equity, and security as the goals of their research program, with the goals specified for both the state and nation. At the other extreme, the goals can be specified in terms of such things as increasing food supply, increasing food supply in efficient manner, producing a more fertilizer-responsive society, developing an insecticide to kill insect X, etc. 3.4 General goals for agricultural research that are somewhat between the two extremes depicted above, and on which there would probably be general agreement, are as follows: - 20 - a. To increase consumer welfare by increasing food supplies; b. To increase income and employment of workers in the agri- cultural sector; c. To increase the total net income of the agricultural sector; d. To maximize the contribution of agriculture to the growth of the economy as a whole; e. To preserve the environment and ecology; and f. To promote rural development in the sense of providing a wider range of opportunities for rural people. 3q5 Perhaps the most important thing to note about these goals is that the attainment of one goal may be in conflict with the attainment of another. For example, the maximization of employment may imply a reduction in efficiency in production, and hence an increase in the cost of food to consumers. Conversely, rapid changes in production efficiency which might imply a decline in prices for consumers, could at the same time imply a reduction in employment. 3.6 The list of potential conflicts could easily be extended. But the point we want to emphasize is that because of these conflicts, actual or potential, there are important trade-offs among the goals. More of one can be attained only at the expense of another. Moreover, something has to be known about these trade-offs before much progress can be made in establishing priorities. 3,7 Another important point to note is that seldom is only one goal relevant, despite the common tendency to describe research programs as if they were single-goaled. A report- on interniationalpriorities recognized this when it discussed explicitly the criteria used to establish priorities.(96). The relevance of the multiple-goaled objective function points up the importance of knowing something about the trade-offs among goals and objectives. It also points up the importance of finding some means of establishing weights for the various goals. 3,8 Finally, the individual goals will vary in their importance accord- ing to the policies pursued by the government and the stage of development of - 21 - the country. 1/ If policy-makers are pursuing import-substituting industrial- ization policies, for example, it makes little sense to channel research resources to export crops. Conversely, if export promotion is a goal of economic policy, research to improve the comparative advantage of a given crop can reinforce the economic policy and contribute to the attainment of larger development goals. 3,9 Similarly, if government policy makes it unprofitable to use fer- tilizer, it makes little sense to invest heavily in fertilizer research. But if the country is well endowed with fertilizer resources and the government is attempting to exploit them, fertilizer research may have a high social payoff. 3.10 Other examples of the relation between goals for the research program and government policy could be cited. However, we turn now to a discussion of the relation of goals to the stage of development of the country. Perhaps the most important point to note here is that the product mix consumed by society changes as per capita incomes rise. At low-levels of development the bulk of food consumption consists of cereals, starchy root crops, and pulses. As per capita incomes rise, the product mix shifts toward animal sources of protein, fruits and vegetables, and more processed foods. These shifts in the composition of demand as the stage of development changes must be considered in formulating goals for the research program. 3.11 At early stages of development countries are likely to foster import-substituting policies. The autarchy that results has important policy implications for agricultural research. With autarchic development policies there is little potential for exports. Hence, agriculture's contribution to development will come through rather different means. 3412 Viewed from a somewhat different perspective, at early stages of development obtaining declines in the price of wage goods such as food can be an important means of fostering industrialization and economic development. At higher levels of development, food makes up a smaller proportion of the consumer's budget, and obtaining reductions in the price of food is relatively less important. At this later stage agriculture may make its most important contribution to development by releasing labor to the non-farm sector. 3.13 Finally, at early stages of development, man/land ratios tend to be relatively high, and an important goal of the research program should be to ease the constraint implied by an inelastic supply of land. At later stages of development, labor will be draining out of agriculture at a rapid rate, and the constraint to output expansion may be a relatively inelastic supply of labor. In this case the goal of the research program should be to raise labor productivity. 1/ For a more comprehensive discussion of this set of issues, see Schuh (83). - 22 - 3.14 In conclusion, goals for the agricultural research program are a function of the mix of economic policies used by the government and the stage of development of the economy. The explicit specification of these goals and knowledge of the trade-offs among them are important bases for assessing and monitoring the productivity and cost effectiveness of the research program. 3,15 This discussion should make it clear that the problem of setting goals for an international research system is rather more complex than for an individual country. What are rather obvious goals for an individual country may not be appropriate for an international institution. Moreover, an international institution may need to differentiate its goals from those of national research organizations, and there may be a need for differentiation of goals among international centers based on the particular parts of the world they attempt to serve. IV. THE SIDE EFFECTS OF TECHNICAL CHANGE 4,1 The output of agricultural research is new technology that is adopted by individuals or firms. The adoption of this new technology may have important side effects in the economy and society. Some of these may be of a positive nature, while others may be of a negative nature. In this section we discuss the nature of these side effects, since they need to be taken account of explicitly in assessing the benefits and costs of agricultural research. A. Income Distribution Consequences 4,2 Technical innovations by definition have resource-saving and re- source-augmenting effects. Hence, they can logically be expected to influence the distribution of income. These effects have been the source of much criticism of technology-based development programs. 4,3 Unfortunately, many of the criticisms have been based on rather superficial and naive interpretations of the data, or on no data at all. For example, it is seldom recognized that the production and distribution of new production technology, by virtue of being a form of human capital, will in the aggregate tend to raise the share of output going to labor, even though particular groups in society may be harmed. Similarly, to the extent that food prices decline as a result of technical change, income is redistributed in favor of the poor in a relative sense since low income groups spend a larger fraction of their budget on food. 1/ 1/ In general, there has been a failure to recognize the benefits to con- sumers of technical change in agriculture. Attention focuses on regres- sive income distribution consequences within agriculture, to the neglect of the progressive effects for consumers. In the final analysis, only empirical research can determine whether the benefits to one group are sufficiently large to offset the losses to another. - 23 - 4.4. At the same time, however, particular groups in society may suffer deleterious income distribution consequences as a result of technical change, especially if the appropriate complementary research and economic policies are not implemented. Some forms of mechanization, for example, can displace unskilled workers, while at the same time generating pressures for farm enlargement which have income distribution consequences. Other forms of technical change can create serious regional income distribution problems, as they give a comparative advantage to one region in favor of another. 4.5 Three points are important about this aspect of technical change. The first is that these side effects should be taken into account in assessing the costs and benefits of agricultural research. The second is that it is not clear a priori whether these side effects will be a benefit or a cost. It will depend on the nature of the technology, plus the conditions of demand. 1/ And third, it is important to identify what groups benefit from technical change and what groups lose, and then relate this to the goals of the research endeavor. B. Employment Effects 4.6 New production technology may have important effects on employment. Although employment effects may be related to changes in income distribution, the employment issue is sufficiently important in its own right to merit separate discussion. 4.7 Biological and chemical innovations in agriculture tend to raise the physical productivity of labor. As long as one ignores the product market effects, this increase in productivity should increase the demand for labor. This, together with the increased labor requirements associated with such innovations, has caused many to expect improvements in agricultural technology to solve the employment problems that plague so many low income countries. 2/ 4'8 Whether it will or not depends importantly on the conditions of demand for the product. If, as is generally the case, a small increase in output leads to a relatively larger decline in the price of the product, this product market effect can outweigh the direct (and positive) productivity effect. Hence, on net, the effect of the technology will be to displace labor. 3/ Such a problem will be especially severe with staple food products, since the demand for these products will be relatively insensitive to price. 4,w. Such employment effects should also be taken into account in assess- ing the costs and benefits of agricultural research. How to account for these effects is not straightforward, however. If the labor so released becomes 1/ For a review of the theory pertinent to answering this question, see Evenson (32). For a more general discussion of the issues and implica- tions of these distribution consequences in the context of rural develop- ment policy, see Agriculture and Rural Development Department (2). 2/ For example, see Johnston and Cownie (50). 3/ Wallace and Hoover (103) found that this was the case in the US. - 24 - unemployed, then it is a cost to the research program. However, the emergence of unemployment suggests that labor markets are either distorted or imperfect. Hence, there is an imputation problem with respect to what portion of the costs should be charged to the research program and what portion should be charged to the distortions and imperfections. The answer to this question requires considerable analysis of the individual instance. C. Environmental Consequences 4.10 The third class of side effects associated with technological change includes the effects on the environment. Some of these effects will be positive, such as those that reduce erosion, and should be charged to the benefit side. Others, such as the pollution of streams and lakes with ferti- lizer and pesticide runoff, may be negative. These, of course, must be charged as a cost to the research program. 4.11 To conclude this section, it should be noted that the various side effects can be quite sizeable, and they may be either a benefit or a cost to the research program. Both of these should be taken into account in monitoring and assessing agricultural research programs, although the most controversy tends to surround the negative side effects. 4,12 It should be noted, moreover, that the existence of the negative side effects is not necessarily cause for not producing and using the new technology, as is often implied. The net benefits may be sufficiently large to provide compensation to those who are harmed and still leave a net surplus. This, in turn, becomes a question of economic policy. With appropriate policy these effects can be minimized or eliminated altogether. V. METHODS AND PROCEDURES FOR EVALUATING AGRICULTURAL RESEARCH 541 This section contains a review of various methods and procedures that might be used to evaluate the effectiveness of agricultural research programs. The material is organized under two headings: (1) procedures used to make an ex post evaluation of the effectiveness of agricultural research, and to evaluate the contribution that agricultural research has made to the economy as a whole, and (2) procedures that have been used to make an ex ante evaluation of research proposals and programs, largely with the goal of developing a more effective means of establishing priorities but also for purposes of justifying budget requests. The examples cited range from broad sectoral studies to narrow evaluations of particular research projects. 5,2 The purpose of the discussion is not to make a critique of past procedures, or to enter into a discussion of the intellectual niceties of particular procedures. Rather, the objective is to provide the reader with a general notion of the procedures that have been used in past studies, the data required to implement them, and some of the issues involved. - 25 - A. Ex Post Studies 5,3 Studies that have attempted to make ex post evaluations of agricul- tural research have for the most part focused on output-increasing techno- logies, and have neglected other contributions of agricultural research. The procedures used can be grouped into five different classes: (1) those which attempt to estimate the resources saved by the adoption of new technology; (2) those which use the concepts of producer and consumer surplus; (3) those which introduce investments in research into an aggregate production function; (4) those which estimate the impact of the technology on national income; and (5) those which have attempted to identify the effect of increased output on nutritional status of the population. We discuss each of these in turn. a. The Inputs-Saved Approach 5,4 Professor Schultz (85) used this approach in what was to the best of our knowledge the first major attempt at quantifying the returns to investment in agricultural research. His interest was in US agriculture as a whole, and hence he made no attempt to consider individual research programs or particular technological innovations. 5,5 If this approach is used to measure resource savings over an extended period of time, an index number problem arises in measuring the value of inputs saved since relative factor prices change over time. To deal with this problem Schultz developed upper and lower limits for the resources saved by in one case using price weights from the early part of the period and in another case the price weights from the end of the period. The resource savings are then estimated by determining how many resources would have been used to produce the output of a base period using the techniques of production of an earlier period. A comparison of this with the resources actually used provides an estimate of the resources saved. 5%6 From this perspective, the value of the resoarces saved constitutes the benefits from the research. The costs of producing these benefits are then estimated by calculating the cost of all research and extension in the country--both public and private. A benefit-cost ratio can then be calculated, or the data can be used to make an estimate of the social rate of return. 5.7 This approach could be extended to individual commodity programs or to more narrowly defined technological innovations. Since aggregate data are not likely to be available in sufficient detail for such an analysis, however, it would be necessary to estimate resource savings either from experimental data or surveys of farms. When combined with data on the extent of use of the innovation, an estimate of the total resources saved could be made. Such an approach would be especially useful for evaluating innovations that are more directly resource-saving than output-increasing. - 26 - b. The Use of Consumer and Producer Surplus 5.8 An alternative approach is to measure the benefits and losses from technical change by means of its impact on what economists refer to as con- sumer and producer surplus. 1/ The technological innovation is assumed to shift the supply curve for the product to the right. When this occurs, consumers benefit from having more of the product available, and producers may benefit from the reduction in costs of production. The concepts of consumer and producer surplus can be used to measure these benefits (and possible losses). 5,9 The advantage of this approach is that it is relatively flexible, and in the hands of a skilled analyst can be modified to take account of a number of side effects of technological change, as well as some indirect effects such as the impact on trade, and the impact of trade and price policy on the distribution of benefits from technical change. The methodology is feasible, however, only when a set of improved technologies can be assoc- iated with a particular research program or programs. If there should be significant technological transfer among countries or regions, or from other research programs, it will not be possible to associate shifts in the supply function with a specific research program. 5,10 The basic analytical framework is illustrated in Figure 1. A shift in the aggregate supply curve (from S to S') for the commodity under considera- tion is depicted, with this shift assumed to be attributed to improved tech- nology. The shift in the supply curve produces a change in the consumers surplus by the area P0ABP1(the area ADB plus the area P ADP1). This is a flow of benefits that arises because consumers are able to purchase more of the product and at a lower price (in the general case). The same shift in supply will produce a change in producers surplus by the area BDO minus the area P ADP . The total change in economic surplus (producers plus consumers) will be the area of AOBo 5,11 The simple framework provides the basis for analyzing various aspects of agricultural research. The empirical information required is knowledge of how much the technical change shifted the supply curve, and knowledge of the parameters that describe the conditions of supply and demand for the product. Information on the costs of the research program required to induce the shift in the supply curve is also required if a full benefit-cost analysis is desired. 1/ For an easily understood explanation of economic surpluses and their use in measuring the returns to agricultural research, see Hertford and Schmitz (48). - 27 - $ S A P~ p~ ~ ~ ~~~ f 0, C] 0~~~~~~~~~~~~~~~~~~~~~ World Bank 18529 FIGURE 1 BASIC MODEL FOR THE ANALYSIS OF CONSUMER AND PRODUCER SURPLUS - 28 - 5.12 One of the most important advantages of this approach is that it provides a means of analyzing how the benefits of the research are divided *-etween consumers and producers. The only thing required for such an analysis _s knowledge of the demand and supply curves. As is readily apparent from the figure, producers can sustain losses from the technical change. All this requires is that the area PoADPI be greater than the area BDOo This distribu- tive aspect is more important if policy makers should have as a particular goal an improvement in the welfare of either producers or consumers0 5.13 The model may be applied to a closed economy or to an economy open to trade0 The demand elasticities in an open economy will, of course, tend to be quite high, meaning that there will be fairly small changes in price associated with changes in the quantity supplied0 Hence, if the technical change occurs for a product that is being exported, most of the direct benefits of the research will go to producers, unless there should be government inter- vention0 1/ Consumers will benefit indirectly, however, since the added foreign exchange which increased exports earn will lower the price of imports and help to finance a higher rate of growth0 These indirect effects should be taken account of in calculating the benefits and costs of research0 5.14 The model can also be modified to take into account price and trade policies, and their impact on the distribution of benefits from technical change0 Schuh (84), for example, has argued that the over-valued dollar in the post-World War II period caused a larger share of the benefits of technical change in US agriculture to be transferred to the US consumer than would have been the case if exchange markets had been free0 Akino and Hayami (3) have examined the rice breeding program in Japan, which was interesting in light of the fact that Japan was a net importer of rice during the period covered by their study0 These authors give explicit attention to the distribution of benefits between consumers and producers, and conclude that in the absence of trade Japanese producers would have been net losers from agricultural research. 5A15 Evenson, et al. (33), consider the distribution of benefits from rice research in the Philippines0 In that case, imports have been utilized to maintain a stable price for consumers, with sufficient rice imported to maintain a target domestic price0 Suppose PI in Figure 1 is the target price0 With the original domestic supply function, the quantity DB would have been imported0 The shift of the supply function to S' would eliminate rice imports. The consumers surplus for rice would remain unchanged, but producers would gain by the area ODB. This area represents a welfare gain to society and is equal to the change in the resources devoted to domestic rice production, OBQ1 - ODQ,, plus the value of the imports in the initial situation, Q'DBQ,. 5q16 These examples show the potential richness of the analytical framework. To the extent policy makers are interested in the effects of economic policies, in the distribution of benefits and costs from policies and I/ For a more detailed analysis, see Castro and Schuh (75). - 29 - technical change, and in trade implications, the model provides a means o- giving them some important answers. Moreover, these answers are quite impor- tant in establishing research priorities, and in managing the process of technical change for the greatest good. 5.17 The model can also be modified in a number of important ways to accomodate different problem situations. For example, Griliches (42), in his first use of the economic surplus concepts to estimate the benefits of tech- nical change, assumed that all the benefits of agricultural research were realized in the form of a consumer surplus. Two alternative estimates of this surplus can then be made, depending on whether the supply curve is assumed to be perfectly inelastic or perfectly elastic (Figure 2). PRICE PRICE S~~~~~~~~~~~~~~~~~~~~~~~~~ QUANTITY QUANTITY World Bank - 18530 FIGURE 2 MEASURES OF CONSUMER SURPLUS UNDER ALTERNATIVE ASSUMPTIONS ABOUT ELASTICITY nF 1I1VPLy - 30 - 5.18 The shaded areas define the alternative measures of the consumer sorplus and are assumed to represent the benefits of the technical change that resulted from the research effort. Griliches argued that these alternative measures constituted upper and lower limits of the benefits of the research, although Lindner and Jarrett (57) have recently taken issue with that con- clusion. Griliches also assumed a closed economy and ignored the potential foreign exchange that might be earned from technical progress. 5.19 The strength of the Griliches approach is its simplicity. It assumes that the price elasticity of demand is -1, and thereby abstracts from general equilibirum or resource adjustment problems. No estimates of either the demand or supply parameters are required, since they are all handled by assumption, and the trade sector and distribution consequences are also ignored. The major empirical problem is to obtain a measure of productivity gain that reflects only the output of research. This measure is required, of course, to know how much to shift the supply curve. 1/ 5,20 Peterson (69) has shown how general equilibrium effects can be taken into account in estimating the net benefits of research. These effects arise by virtue of the changes in resource productivity associated with technical change, which in turn cause resources to be either induced into the progressive sector, or expelled from it. Peterson's procedure is based on the simple fact that if the price elasticity of demand were equal to -1, the total value of any price-quantity combination along the curve is the same. The relation between the unit elastic demand curve and the actual demand curve provides the means for taking account of the general equilibrium effects. 5,21 Consider Figure 3. Assuming that we shift the supply curve back to the left to make the evaluation, the decline in output q"q' is due to a decline in productivity of a fixed bundle of resources if the technology were withdrawn. As a consequence of this decline, net social benefits decline by q"q'BB'. (These net social benefits are made up of both producer and consumer surpluses.) The decline in output represented by q q" is due to a shift of resources out of the sector. These have an opportunity cost (as represented by the area under the supply curve) of q q"G'G. But there is also a loss in consumer surplus in the amount of q q"B'8. Therefore, the net gain due to the liberation of resources is G B' G'. This amount has to be subtracted from the estimate of net social benefits measured in the partial equilibrium framework to take account of the general equilibrium effect. 5,22 In the particular case considered, the actual demand curve was assumed to have an elasticity greater than I in the relevant range. If it had an elasticity less than 1, the relationship between the two demand curves would be different, and the triangle would represent a net loss, and have to be added to the partial equilibrium estimate of benefits. I/ Benefits are always measured by assuming the production technology were withdrawn (a shift to the left of the supply curve) in order to provide a conservative estimate of the benefits. - 31 - PR ICE S / /~~~S. -------- -- ----- X/ __ _ _ ----- qo q q QUANTITY FIGURE 3 ILLUSTRATION OF GENERAL EQUILIBRIUM EFFECTS ASSOCIATED WITH SUPPLY SHIFT World Bank -18531 - 32 - 5.23 The Peterson formulation of the problem is somewhat demanding in terms of the measures of economic parameters required to make the estimate. By the same token, however, a great deal more can be learned from the analysis in terms of the distribution of benefits between consumers and producers and the adjustments for general equilibrium effects. 5.24 Schmitz and Seckler (82) extended this basic model in an interesting way. As noted above, the increase in productivity due to technical change can require the release of resources from the sector, depending on the conditions of demand for the product. If alternative employment possibilities are not available, the resources released may be unemployed. If they are, then the income lost by these unemployed resources has to be deducted from the benefits of the technical change to determine the net benefits. 5,25 Schmitz and Seckler viewed the mechanical tomato harvester as an in- novation of agricultural research which caused farm workers to be unemployed. They first estimated the benefits of the research on the harvester in the usual way, and then estimated the returns the unemployed workers would have received in the absence of the technical change. These sacrificed returns were then subtracted from the benefits. 5,26 This procedure amounts to taking into account the adjustment cost associated with the technical change. It should be noted, however, that resources could be induced into the sector as a result of technical change. If this occurs, it should be taken into account as well, since it will mean a reduction in output in other sectors. 5,27 Ayer and Schuh (10) further extended the model by specifying a cob- web behavioral model for the sector. This model leads to a different kind of adjustment costs because ex post results do not square with ex ante expecta- tions. Their procedures net out these adjustment costs from the net flow of benefits. 5.28 Ayer and Schuh also made a qualitative analysis of which group of factor owners received the benefits of the technical change. This was done by taking into account the characteristics of demand and supply for the individual ,input categories. They also showed that trade and other economic policy played an important role in determining how the benefits of the technical change were distributed between consumer and producer groups. 5,29 The final contribution to the methodology using economic surpluses was made by Hayami and Peterson (46). Although the model will not be elabora- ted here, their analysis is important in that it provides a means of evaluat- ing economic research and research of a more general nature than that con- cerned with production alone. Their study was concerned with estimating the social returns to government expenditures on public information services such as the collection and reporting of information useful for decision making in both the public and private sectors. Their particular interest was the statistical reporting of US farm commodities. - 33 - 5.30 The basic assumption of their model was that erroneous information causes producers to make erroneous production decisions and also distorts optimal inventory carryovers. Hence, marginal improvements in the accuracy of these statistics reduce the social cost of misinformation, which in turn can be considered as an increase in net social welfare. By relating the marginal improvements in the net social welfare to the marginal cost of providing more accurate information, they can estimate marginal social benefit-cost ratios for the various levels of accuracy of the information. 5,3J We believe this basic formulation of the problem has more general application. Agricultural research in its broadest sense is in effect nothing more than providing information to relevant decision groups. As the evaluation of research moves beyond its past concentration on output-increasing production research, we believe the Hayami-Peterson model points the way to a more general methodology. 5,32 To conclude this discussion of methodologies using the economic surpluses concepts, we note that Lindner and Jarrett (57) have recently made a critique of the various assumptions used by previous authors. Since their paper deals with some of the finer parts of the analyses, we pass over them here.- c. The Production Function Approach 5,u33 A completely different approach to estimating the payoff to and the effectiveness of agricultural research is to specify the production function for the commodity or the agricultural sector as a whole in a sufficiently broad framework that social inputs such as agricultural research can be included in it. The conventional production function includes only on-the- farm inputs. But inputs provided by the public sector can (and perhaps should) be included just as well. 5 34 Griliches (43) was perhaps the first to use this approach, but it has also been used by Peterson (69), Evenson (31) and others (55). The advantages of this approach are that it effectively controls for the other inputs used in production (if the data are measured accurately) and provides an estimate of the marginal product of research. (Methods referred to above all give estimates of the average return.) A marginal return is more useful than an average return to decision-makers studying the merits of new research projects. 5,35 An important contribution of the Evenson analysis was to throw light on the time path of response to increased expenditures on research. He found that the returns over time first increased and then decreased, with the high point occurring after about six years. Estimates of the rate of return to expenditures on research are senBitive to t-he time path in which benefits are forthcoming. The insights gained by the Evenson analysis can be used to improve the estimates of the rates of return. - 34 - 5,36 The production function approach is potentially as rich as the economic surplus approach, although its flexibility lies in somewhat different directions. For example, attempts have been made recently to refine this approach so that it could be used for decision-making purposes. Bredahl and Peterson (16) estimated an aggregate production function for US agriculture using 1969 state data as observations, and included expenditures on agricul- tural research as an explanatory variable. They were interested in comparing the productivity of research among cash grains, poultry, dairy, and livestock. Their analysis provides estimates of the marginal rate of return to incremental changes in the investment in research on each commodity group, plus an esti- mate of the marginal rate of return for each state. Hence, their results provide a guide as to productive reallocations of research expenditures both among commodity groups and among geographic areas. 1/ A similar analysis might be done using country observations rather than state observations. 5237 Evenson, Flores, and Hayami (33) have used the basic production function approach to deal with the technology transfer problem. Their interest was in analyzing the costs and benefits of rice research, a case where both an important International Center and national research programs have produced new knowledge which presumably differs in its transferability. They also attempt to take account of knowledge transfer from other disciplines. 5,38 The basic model amounts to regressing the change in yield of rice over a base period on a set of farm input variables that would be expected to influence yields plus a set of knowledge stock variables that are constructed as cumulated research investment. The knowledge stock variables include (1) research undertaken in agronomy and plant breeding specifically to improve rice technology; (2) research activity in plant physiology, phyto-pathology, and soil science (research which is not commodity specific); (3) agronomic and plant breeding research activity in countries other than the country in question, but which are in the same geo-climate region; and (4) agriculturally related scientific research in other countries located in the same geo-climate zone. 5,39 The regression analysis provides a means of statistically isolating the effects of these various research programs, while at the same time control- ling for the use of other inputs that are expected to influence observed yields. The benefits of research can then be imputed to particular research programs, and one has the basis for answering allocatory questions. In principle, increments in budget would be allocated to where the social rate of return were highest. d. Impact of Research on National Income 5,40 Tweeten and Hines (100) have used a still different approach to the evaluation of the returns to agricultural research. Their approach is somewhat 1/ Robert Thompson used similar methodology in his Ph.D. dissertation to estimate the marginal rates of returns to research activities by state in Brazil. See Thompson and Schuh (99). - 35 - similar to the input-saving methodology, and recognizes explicitly that a contribution of new agricultural technology is the resources it releases to the nonfarm sector. 5.41 Tweeten and Himes calculate how much lower the national income would be if the percentage of people on the farm were still the same as in 1910 and the resulting additional farmers had the income of today's farmers instead of today's non-farmers. This provides a measure of the benefits from research. Then they estimate the costs of public and private research, education, and federal programs, and use these to calculate a benefit/cost ratio. 5.42 This approach provides only a rather crude approximation to the benefits of agricultural research. It does provide it in a form understand- able by policy workers, however, and may be feasible when lata are rather scarce. e. Nutritional Impact 5,43 Research goals for agriculture may be expressed in nutritional objectives. Pinstrup-Andersen, Londono, and Hoover (72) have developed a procedure to estimate the nutritional implications of alternative commod'ity priorities in agricultural research and policy. Their model estimates the distribution of supply increases among consumer groups, the related adjust- ments in total food consumption, and implications for caloric and protein nutrition. This procedure permits a translation of increases in agricultural output to its impact on nutrition, and by income groups. Hence, equity and nutritional considerations can be analyzed. 5,44 This model is rather demanding in terms of detailed knowledge of demand parameters and present consumption patterns. It does not in itself provide estimates of rates of return, but could be extended to such an analysis if nutritional objectives were translated into a suitable form. This approach does provide information that could be of considerable value in establishing research priorities if improved nutrition is the research goal. * * * 5.45 To conclude this section, we would note that a rather rich set of research procedures have been developed whereby research can be evaluated and its contributions and various effects analyzed. It should also be rather obvious that there is no one approach that offers a panacea. Different approaches are useful for answering different questions, and the particular question will vary a great deal depending on the individual problem situation. 5,46 The main conclusion we draw is that there is an ample methodological base for using data generated by the economy and past experience to understand and analyze the consequences of investments in agricultural research and its interactions with economic policy and institutional arrangements. The knowledge gained from these analyses can in turn be used to improve the decision-making with respect to scarce research resources. - 36 - B. Ex Ante Models 5.47 In this section we review some of the major procedures and methods that have been used to improve en ante decision-making with respect to research resource allocation. The literature in this area is vast, since ouch procedures are widely used with industrial and military research. I/ Our review concentrates on the models developed for agricultural research0 These models range from approaches which provide a systematic means of utili- zing informed judgment, to approaches which attempt to provide empirical knowledge on the consequences of alternative causes of action. 5,48 Before discussing some of the methods suggested in the literature, however, it is fitting to recall the words of Cetron and Johnson (22): "We are well aware of many of the omissions and weaknesses of these quan- titative-selection or resource-allocation techniques. It should be stressed again that they are not intended to yield decision, but rather information that would facilitate decision. Indeed, these techniques are merely thinking structures to force methodical, meticulous consideration of all factors involved in resource allocation. Data plus analysis yield information. Information plus judgment yield decisions." 2/ And they go on to say: "It is wrong to say that one must select intuitive experience over analysis or minds over machines; really they are not alternatives; they complement each other. Used together, they yield results far better than if used individually." 5,49 Pure analysis or pure intuition should not be the objective of any decision-making group.- There is an unknown optimal mix of analysis and intuition. This optimum combination cannot be defined a priori and generally. It has to be reached through the joint effort of analysts and managers. A continuing, sequential, dialetical process will be necessary to determine what is necessary and possible with respect to Loasby's (58) "width of agenda", "set of control variables", and "degree of programming." 3/ 5,50 Decision problems such as those associated with the allocation of resources to agricultural research have been classified according to three major criteria. The first is according to time, with the problems classified according to whether they are static or dynamic. For decision I/ For an analysis of the state of the arts as of 1968, see Rubenstein (78). He calls attention to the fact that up to that time much of the literature was the result of laboratory experiments, not models that were actually implemented, and that to be useful to the research manager the entire process (problem recognition, research, development and testing, engineering and tooling, adoption and continuous improvement) had to be carried out. See also Rubenstein (77), Cartwright (20), Augood (9), Clarke (24), Gear (37), Cetron, et. al. (23), Caty, et. al. (21), Shumway (88), Allen (6). 2/ Underlined words were stressed in the original. 3/ See below for more detail. - 37 - models formulated in a static framework, time is not considered explicitly. The actions and reactions involved in these situations are either assumed to take place instantaneously, or without a time dimension. For models formulated in a dynamic framework, time is considered explicitly and the actions and reactions are assumed to occur at different distinguishable points in time. 5,51 There is also a compromise between static and dynamic models, the comparative static framework. In this case a static analysis for different points in time is made, and then a comparison is made of the results over time. The objective is to keep the simplicity of static models while gaining some advantages of the dynamic perspective. 5 52 A second criterion according to which decision problems may be classified is according to the degree of uncertainty I/ involved. At one extreme, models have been built assuming perfect knowledge with respect to all events taking place. It is assumed that the decision-maker has complete certainty about actions and resultant outcomes, both qualitatively and quan- titatively. These models are called deterministic. Against these simpler models, there are models that try to take into account the realities of uncertainty facing decision-makers in any area of endeavor. These are called probabilistic models, since they try explicitly and formally to consider the fact that certain variables have a probability distribution of possible values and are not under the direct control of the decision-maker. The most the decision-maker can do is to insure himself against the risk involved in this situation. As with any insurance scheme, premiums and the probability dis- tribution of losses will have to be evaluated. An extreme case of uncertainty is when the probability distribution of the uncertain events is not known. 2/ 5f53 The third criterion according to which decision problems may be classified is according to the "environment" in which the decision-maker, be it a person or a group of persons, has to take decisions. To the extent that there is not an intelligent and informed force reacting to the decision-maker in a competitive way, i.e., to the extent there is no conflict of interests between the decision-maker's decisions and the opponent force's objectives or goals, the decision problem is represented by a one-decision-maker model. Otherwise, conflict of interest between competing decision-makers, persons, or groups of persons, results in the necessity of the decision model taking into consideration each opponent decision maker's reaction to one's decisions. These much more complicated decision situations are analyzed through models generally designated by two- or n-decision-maker models. 3/ 11 No effort is made here to distinguish carefully between risk and uncer- tainty in Knight's (54) sense. A discussion of the distinction between risk and uncertainty may be found in Roumasset (76). 2/ A discussion of risk and uncertainty has been part of the literature on decision theory for many years. A standard reference for these concepts and for the discussion of subjective versus objective probability may be found in Luce and Raiffa (59). See also Hampton et al. (44). 3/ Again, Luce and Raiffa (59) is a good reference. - 38 - 5.54 Most of the decision problems in the area of agricultural research resource allocation may be treated as a one-decision-maker situation and therefore do not call for models that represent conflict of interests. But both time and uncertainty are important elements in agricultural research. Moreover, to obtain the possibility of explicitly considering time and un- certainty in analytical models of research resource allocation, a price has to be paid in terms of mathematical complicationo. 5.55 The degree of complexity will also depend on other factors. Loasby (58) classifies decisions according to three aspects: (1) width of agenda, (2) specification of the set of control variables, and (3) degree of programming. The width of agenda refers to the definition of the system boundaries. The decision to consider single or multiple goals, as well as single or multiple restrictions, is a relevant issue in this context. The specification of the set of control variables had to do with the planning horizon or, as it may be put, with the length of the run,, hether short or long. The degree of programming depends on how precisely the decision pro- cedures are prescribed. It is clear that in modeling decisions, the degree of difficulty will depend on how broad the system's boundaries are, how many variables are considered simultaneously, and how well programmed the whole procedure is. a. Scoring Models 5>6 Scoring models are relatively simple procedures to formalize the decision process involved in the choice of a research portfolio. Key eval- uators, usually the scientists themselves, are called upon to express their evaluation of alternative research projects. These evaluations are based on the potential contribution of each research project to a prespecified goal or set of goals. These goals can be measured in a continuous or discrete way, but evaluations are expressed numerically. Where more than one goal is involved, the same or other evaluators will have to establish a weighting structure. That is, they will have to express numerically the relative importance to society of each goal, especially if they are competitive. Complementary goals can be reduced to only one goal. 5.57 Scoring models are generally very flexible not only with respect to the number of goals, but also with respect to the type of goals considered. The specific approaches we will review here are those developed by Iowa State, North Carolina and NASULGC-.USDA. (i) The Iowa Model I/ 5.58 The primary purpose of the Iowa model was to ensure the greatest return for the research money spent at the experiment station. 2/ In addi- tion, however, an increase in the value of research output was expected due to better evaluation and selection of projects, and an increase in resources for research due to an improved ability to demonstrate their efficient use. I/ An algebraic representation of this model is presented in Appendix A. 2/ See Mahlstede (60). - 39 - 5, 59 A first and very important step in the implementation of the Iowa procedure was a decision by all administrators involved regarding the neces- sity for a more formal method of resource allocation, and their agreement on and commitment to a proposed scheme for evaluating research projects. The next step consisted of developing a set of goals and sub-goals. Three goals were selected: growth, equity, and security, which were assumed to apply to the state of Iowa as well as the country as a whole. 5.60 The first experiments with the model considered only growth. This was due to the difficulties involved in weighting the three goals. Nevertheless, this review will be concerned with the conceptual model in which all three goals were considered. 5t6l The research effort of the station was divided into three major areas: commodity research, resource research, and agricultural management research. These three areas were further subdivided into 19 sub-areas com- prising one or two products, or resources, or aspects of agricultural and management research. To each one of these 19 areas a panel of experts was assigned. 5 62 The panels were then asked to identify all research alternatives in each area. Each panel member had to present a list of research projects that in his opinion would represent a significant contribution to knowledge and to the goals of growth, defined as value of resources saved and as value of increased output. The panel had to consolidate these individual opinions into a list of suggested research activities. In addition, for each alter- native research project suggested by the panel, an estimate of the cost in terms of science man-years and other supporting costs was presented. 5 63 As a second step in the procedure, these lists of suggested research activities were submitted to another special panel for evaluation in terms of the second criterion, equity. The panels were asked to give an evaluation of each project in terms of its contribution to absolute and relative equity. This procedure was then repeated in order to take security into account. 5,64 A scoring procedure, where each project receives a "grade" on a given scale according to its contribution to growth, and then to equity and security, is the core of the method. The difficulty in scoring is evident, as well as its strong dependency on the experience and wisdom of the panel members. 5q65 This is a static, deterministic model insofar as time and uncer- tainty are not explicitly accounted for. The effect of some kinds of un- certainty can be evaluated by means of sensitivity analysis, however. 5,66 A question might be raised as to whether this model requires too much guessing. The answer is probably yes, but they can be informed guesses, which tend to improve with repeated trials. A second question is whether - 40 - this model is better than no model at all. Again the answer is probably yes, so long as scientists and decision makers work together to improve itoI/ (ii) The North Carolina Model 5,67 The major question addressed in the North Carolina Agricultural Experiment Station model was how much emphasis (in terms of resources) should be put into each of the research problem areas. 2/ The procedure involved several groups of interdisciplinary research and extension faculty, plus several groups of external scientists. Groups of administrators from the experiment station and from the departments were also involved. 5,68 The interdisciplinary teams of researchers and extensionists were allocated to the following research problem areas: biological sciences and technology, animals and plants, environment and natural resources, and food-fiber-people-economics. They reviewed the entire research program and prepared recommendations on how and when human and monetary resources should be reallocated. Then they rated each of their recommendations according to criteria such as the extent to which the research met state experiment station, department and national goals, the urgency of the problem, cost, relevance, likelihood that research results would not be available elsewhere, and potential contribution to knowledge. 3/ Not all of the above criteria were used in each of the four areas into which the research program was classified. However, a weighting system was developed for the criteria through use of a Delphi procedure 4/ involving the department heads and the administration of the agricultural research station. 5.69 The recommendations of the interdisciplinary in-house teams were then submitted to several smaller groups of external scientists. After reviewing these recommendations, the external groups of scientists developed their own set of recommendations for resource reallocations within the research program, and rated the recommendations of the in-house teams. 5.70 The recommendations of the interdisciplinary in-house teams were finally rated according to the criteria discussed above by each member of 1/ A version of this model was used in the recent World Food and Nutrition Study conducted by the National Academy of Sciences (64). The individual panels were urged to make quantitative estimates of the expected costs and benefits from the priority lines of research. 2/ See Shumway (89) for a detailed report on the North Carolina procedure. See USDA (102) for a description of the classification of research problem areas. 3/ These criteria were adopted from the criteria recommended in USDA (102), See also Williamson (106). 4/ On the Delphi method see Dalkey and Helmer (25), Beattie and Readerr (14), and Brown and Helmer (18). - 41 - three groups: in-house scientists, external scientists, and department heads. Research area scores were then computed by an algebraic formula9 1/ averaging over the criteria all the partial scores attributed to the recom- mended increase in resources to a given area. Weights representing the relative importance of each criterion were used to arrive at a weighted score. These scores represented the evaluation of individual scientists and administrators of the importance of each research area. 5.71 The average score was then computed for each research area. This average score was a numerical expression of the "average" opinion of in-house and external scientists, plus administrators, with respect to the "relative worth" of each research area, given the criteria set that was developed independently. 5.72 The North Carolina method, like other scoring methods, has the advantage of forcing all people involved to spell out formally what they think each research effort will contribute to given goals while at the same time respecting some restrictions. A feature of the North Carolina model that deserves special attention is the computation of two related measures to rank the research areas: the "average" and the "average minus one standard deviation." The "average minus one standard deviation" measure for research areas with the same average score will give preference to the research areas in which opinions were closer together. That is, it will give a higher rank to research areas where consensus is greater. 5.73 Another point to note 2/ is the diversity of opinions demonstrated by the North Carolina effort. The degree of (linear) association among scores given to different research areas by any of the three groups of scorers involved was low (the highest was 0.45). The same low association was found within groups of scorers. 5.74 The major fault with the North Carolina model, aside from those due to the intrinsic characteristics of scoring models, was the failure to specify the goals more precisely. Consequently, each scorer could have a different idea about the goals of the experiment station, the departments, and the country. (iii) The NASULGC-USDA Model 575 The National Association of State Universities and Land Grant Colleges and the USDA jointly defined and implemented a systematic procedure for evaluating and allocating resources to agricultural research. 3/ A task force was assigned to the study and its first step was to prepare a 1/ See Shumway and McCracken (91). 2/ This point is discussed by Shumway and McCracken (91). 3/ The basic document reporting this effort is USDA (102). See also Bayley (13). The present discussion closely follows Williamson's (106) excel- lent presentation. - 42 - general scheme for the classification of research. Research was classified into three major headings, according to whether it was related to an activity, a commodity or resource, or a field of science. The short term objective of the classification scheme was to assemble information to describe research programs and to project needs and priorities for future research. The longer term objective was to develop an information storage and retrieval system. 5.76 A second task in the general procedure was to classify according to this scheme all research going on at the Department of Agriculture's research units and at the state agricultural experiment stations. Information was also developed on the number of scientist-man-years and funds expended during fiscal year 1965. This permitted an estimation of scientist-man-years and funds being devoted to each research category in the classification schemes. 5.77 The third task was to establish national goals for agricultural research. Usually, national goals are too broad for the relationship between them and the agricultural research output to be evident. 1/ It was at this stage of the study that the Planning Programming Budgeting (PPB) 2/ system was adopted by the Department of Agriculture. As Schich (81) says, "One of the major aims of PPB is to convert the annual routine of preparing a budget into a conscious appraisal and formulation of future goal and policies.' Then, the study and PPB had to be adapted to each other in terms of the study's goals and the PPB's "missions". 3/ 5.78 The research program was then divided into six areas: (1) soil, water and air; (2) forestry; (3) horticultural crops; (4) field crops, (5) livestock and poultry; (6) agribusiness and human resources. To each of these six areas a review panel was assigned consisting of people from universities, federal and state agencies, private research organizations, producer groups, industry, and members from the original group responsible for the study. The objective of these review panels was to go into the details of 1/ Mahlstede's (60) discussion of the Iowa model presents examples of how to relate social goals like People's Welfare to intermediately defined research lines. His example is not a general solution, though. See also Pinstrup-Andersen and Franklin (71). 2/ According to Puterbaugh (73) the PPB system implied the use of a problem- - oriented budget, a "zero-base" budgeting process in place of the usual incremental budgeting, and multi-year budgeting. All these aspects involved difficulties. 3/ It is interesting to note Williamson's (106) comment that "Final agreement on the goals was not reached until the estimate of future research needs was essentially complete." - 43 - each area, subdividing them into research problem areas and providing estimates of the number of science-man-years needed in the future. 1/ b. The Minnesota Model 5.79 MARRAIS is classified as a multi-dimensional ranking method. 2/ It is a well thought out logical structure that takes into consideration many of the uncertainties involved in the prediction of costs and benefits in research. 5.80 Three major steps are involved in MARRAIS: specification, esti- mation, and analysis. A fourth step would be selection of the research portfolio, but this is not within the scope of MARRAIS itself. Selection is left to the decision-maker. MARRAIS is an information gathering and processing device to help decision-making, just as most of the other existing methods for resource allocation analysis attempt to be. 5.81 In the specification phase, the alternative research projects to be analyzed and evaluated are defined under lines of administrative respon- sibility. (MARRAIS cannot help in identifying the research alternatives.) In addition, the form of the research results and the unit of measurement of these results are specified at this stage. MARRAIS uses percentages of total objective achievement as the unit of measurement. This is to take account of the fact that benefits may be derived from research efforts that do not fulfill their objectives. Another point to be specified in this stage is who is going to provide all the necessary estimates related to research costs and benefits. The choice of "estimates" has no sound theory on which it can be based. 5.82 The second phase in MARRAIS is estimation. To understand the estimation phase, it is necessary to understand the logical structure of the model. 3/ MARRAIS works either with a present-value formulation of benefits and costs, deriving from these benefit-cost differences (B-C) and ratios (B/C), or with an internal rate of return (IRR) formula. All the estimates are made under alternative levels of average annual expenditure and expected true horizons for project completion. 5.83 MARRAIS recognizes that in practice a lot of uncertainties affect the discounted present value of costs and benefits, so they should be thought of as stochastic variables with given probability distributions. Moreover, the discounted present value of benefits depends on certain variables. MIARRAIS hypothesizes that it depends on the annual benefit accruing from the research, assuming 100 percent adoption of its results, on the adoption patterns over time, on the "scrap" value of certain research facilities, and on the so-called "process" value of research (the increase in the value of 1/ The "future" at that time was the current 1977 year. 2/ See Shumway (88). For a "best informed" exposition on MARRAIS, on which the present discussion is based, see Fishel (36). 3/ An algebraic representation of the model is presented in Appendix B. - 44 - participating scientists plus increased human capital from graduate training involved with the research effort). On the other hand, MARRAIS hypothesizes that the stochastic (discounted) present value of costs depends on the average annual expenditure on research, on the maximum annual expenditure on dissemina- tion, and on the time path of dissemination costs of research starting one period after the project is completed. It should be noted that both the adoption patterns over time and the time path of dissemination costs are functions of time. 5.84 Groups of experts 1/ provide estimates for the variables. Given an average annual expenditure on a project, they estimate the probability of the project being completed in alternative periods of time. Then, with estimates of the mean time to complete the project and the average annual expenditure on the project, benefits are estimated. This results in a pro- bability distribution of benefits from a given project, funded at a given level of annual expenditure and taking the mean expected time for completion. This probability distribution of benefits is weighted by the probability distribution of technological feasibility for each project and funding level. By a random sampling from the distributions of the involved stochastic vari- ables, plus single-value estimates of the non-stochastic variables, estimates are made of the distributions of the difference between benefits and costs, of the benefit-cost ratios, and of the internal rate of return. 5.85 MARRAIS is a relatively sophisticated, multiple-dimension ranking m.del. There may be serious difficulties in applying it in an international context due to the large degree of variation to be found in the relevant variables and the difficult task "estimators" would face. On the other hand, these same conditions are an argument in favor of a model such as HARRAIS. If the difficulties are recognized, an attempt can be made to deal with them in a systematic way. Moreover, the potential for sensitivity analysis which MARRAIS offers with respect to all these uncertain phenomena may be very useful. The use of different parameters for the distribution of the stochastic variables, or the use of different probability functions, are some of the ways of dealing with the precarious confidence that is usually put on such "subjective" distribu- tions. But it may be very expensive both in time and money. c. Pinstrup-Andersen and Franklin Model 5.86 The model developed by Pinstrup-Andersen and Franklin (71) is an attempt to reflect their argument that "concurrence between the technology specification received by the scientist and the technology which results in maximum contribution to the achievement of social goals is the respon- sibility of research management". This concern leads them to analyze the problem of defining working objectives for research from stated national development goals. It should be kept in mind that they are concerned with the allocation of agricultural research resources in developing countries. This helps explain the greater emphasis put on the relationship between development goals and agricultural research relative to other studies. 1/ See Shumway (88) for a discussion of group procedures. - 45 - 5.87 The authors argue that after the identification of the changes in product supply, input demand, and farm consumption necessary to attain the development goals, the identification of the research problems should be made independently of the alternative technologies that can contribute to the solution of the problem. They call this a "technology-free" specification of the problem, since it does not presume ex ante a particular technological solution to the problem. 5,88 This is a very important aspect of the problem definition phase in scientific research. It relates to the "form" in which the problem is pre- sented. The advice is not to jump from farm problem identification to research problem definition in terms of required technology without a careful evaluation of all technological solutions available. Another point to keep in mind is that definition of the problem at the farm level is not an easy task. The work of Hayami and Ruttan (47) makes it clear that low production per acre in one region relative to an-ther region does not necessarily imply a problem in the first region. The differences in productivity may reflect a difference in factor proportions, which in turn are induced by differences in relative factor prices. 5.89 After the identification of problems in a "technology-free" manner and, subsequently, of the alternative technologies available to solve each and every problem, it is necessary to estimate the time, costs, and probabili- ties involved in research and in farm adoption for each alternative technoloRv. The next step is to estimate the impact of the research alternatives on farm consumption, product supply, and input demand. This effort requires some previous knowledge about the economic structure (parameters) of the relevant production sectors and of the product and input markets. 5.90 Then the estimated effects on farm consumption, on product supply, and input demand are used to obtain an evaluation of the contribution of the alternative research lines to the achievement of the development goals. This leads to a specification of the working objectives for the research and of the desired technology. 5.91 The representation of this approach by means of mathematical equa- tions is not simple, since it is a system approach to the allocation of research resources. Moreover, it tries to relate specific research problems to overall aspects of growth, equity, and security through consideration of variables such as income distribution, nutrition, demand for labor and other services, farm consumption, capital formation, supplies and demands, net revenues, risk, etc. A flow diagram usually gives an easier pictorial- understanding of the interrelationships involved, and such a flow diagram can be found in Pinstrup-Andersen and Franklin (71, p. 424). 5.92 Eight social goals are considered, with the objective being to obtain an improvement in the following variables: growth, income distri- bution, employment, farmers' net income, farmers' cash inflow, human nutrition, degree of self-sufficiency in food, and foreign exchange earnings. Other goals might be considered, additionally and/or in substitution of the eight cited, but the interrelationships and parameters in the model will have to be modified accordingly. - 46 - 5.93 This model can be applied to the problem of resource allocation in single product or production factor research (as most of the centers in the international agricultural research system are organized), multi-commodity research, and farming systems research (such as with small farms). Some promising empirical results have been reported by Pinstrup-Andersen and Franklin. 5.94 Pinstrup-Andersen and Franklin's model is an example of a useful effort in the area of research on research. and deserves further testinA and development. It also involves a relatively complicated methodology, however, if research management does not want to invest significant human resources in the problem of resource allocation. As a guide to the kind of information useful to the analysis of resource allocation, the model is quite useful. Its claims that the effects of research are highly dependent on the kind of public policy being pursued by the country is important and in line with points made earlier in this paper. d. Cartwright Model 5.95 The model developed by Cartwright (20) focuses on the allocation decisions for research within a department of agricultural economics. Two decision problems are analyzed: the choice of research areas to work in and the choice of a research job portfolio. 596 A large quantity of information is fed into the model of research and decision. The information includes (1) the amount of time each researcher has to input into each alternative procedure available to undertake the new research areas in a given period of time (quarter); (2) the amount of time each researcher could divert from previous assignments to a new category of employment in a given quarter; (3) the amount of funds of a given category that the new research area would bring to the department in a given quarter if a given procedure were adopted; and (4) the number of new staff positions in a given category that would be created in a given quarter if a certain procedure were adopted. With this information and a formal statement of the staff preference function, the allocation problem is cast in the form of a non-linear integer programming problem. 5.97 Generally, information in (1) and (2) above is not readily available and thus requires some additional estimation by the staff. Information in (3) and (4) is generally available in any proposal for undertaking a new research area or for reallocating resources among current research areas. 5.98 The job portfolio selection model assumes that a more centralized decision procedure is implemented in the academic environment. It involves a simultaneous evaluation and selection of all research jobs once each quarter. Research is classified into four classes according to whether they are dissertion research or not, and whether they are in the agricultural experiment station program or are funded by outside research contracts. A fifth class, of currently active jobs, is also explicitly included, subject to a decision of continuing or stopping these jobs. A job can be modified with respect to time horizons, resource use, etc., by terminating it and at the same time starting a Job with the required modifications. - 47 - 5.99 The information required for the model includes (1) the amount of time required from each staff member by each alternative job in a given quarter; (2) the annual expenditure, by class of expenditure, from each budget component, required by each alternative job; (3) the amount of time uncommitted, by researcher and quarter; (4) and the annual budget, by ex- penditure class and by budget component. Again, staff preferences have to be evaluated in a tentative effort to define the goals and, eventually, an objective function. This is an important and difficult task in the procedure. 5-100 The models for research-area decisions and for job-portfolio decisions are structurally very similar. Cartwright (20, p. 151) calls attention to the following five common characteristics: (1) the decision variables are interdependent and linearly related, (2) these variables can- not assume negative values, (3) at least one variable can take on only one of two values and hence is not continuous, (4) the objective function con- tains several goals (some of which may not have a natural or obvious measurement) with an unknown mathematical form, and (5) the model can assume certainty about all facts involved or it can assume that risks and uncertain- ties are present. 5-101 The solution procedure for the model is not straightforward. An optimization procedure may be used, as well as one that although not finding an optimal solution, uses stimulation to indicate some alternative, accepta- ble solution. 5.102 Cartwright's model is very imaginative. However, this is an example where more "development" of the model would be necessary before its use in routine decision making would be practical. Further development of the goals-preferences-objective function procedure is necessary, as well as of the solution procedures. Also, since it was cast in the framework of a department of a univeristy, a large number of modifications would be necessary to adjust this model to the International Agricultural Research Centers' conditions and needs. 5.103 A short test of a research monitoring and reporting system (REMORS) is also reported by Cartwright (12). This system consists of a register of all current and planned research, plus a system of reporting research progress. Reviews of the "cost-benefit" of REM0RS by staff members were mixed. Of course, they probably had a very short-run review of the procedure. A long-run view, taking account of expected improvements due to continued testing and development, might be more favorable. 5.104 The general conclusion we reach is that although Cartwright's work is to be commended for its rigorous and comprehensive focus on quantative methods for resource allocation, it did not go very far into the development stage, even in the special environment of an academic department for which the model was designed. Therefore, it does not seem appropriate for the inter- national agricultural research system at this moment. e. Castro and Schuh Model 5'105 The major characteristic of the Castro and Schuh model is the emphasis it gives to both growth and distributional effects of research - 48 - and the resultant technological change. 1/ Using the concepts of producer's and consumer's surplus, functional distribution of income, and two-sector models of general equilibrium, Castro and Schuh use analysis and information on key parameters to assess the effects of research and technological change on given products of a given country. 2/ They establish four goals for a research program: (1) to increase the net income of the agricultural sector, (2) to increase employment and income of workers in the agricultural sector, (3) to increase consumer welfare through lower real food prices, and (4) to maximize the contribution of the agricultural sector to the growth of the overall economy. 5.106 Castro and Schuh's model is not a formal, mathematical model, although it could be used as a starting point to build such a mathematical model. 3/ The important point to note is its focus on both the growth and distributional effects of technological change, and on both the direct and indirect effects of research. 4/ 5.107 The explicit consideration of the distribution of benefits and losses between producers and consumers and among factors of production is a desirable characteristic of Castro and Schuh's procedure. 5/ Also important i.s the following up of the effects and reactions of technological change in the agricultural sector into the non-agricultural sector. 5-108 This model minimizes the burden put on scientists and administrators in terms of the amount of information necessary and in terms of the amount of difficult estimates and/or "informed guesses" required. The model depends primarily on secondary data, and the burden of the analysis rests with the analysts. 1/ This model is discussed in Castro and Schuh (75). Details may be found in Castro (74). 2/ Cotton, sugar cane, rice, corn, edible beans, and manioc in Brazil. 3/ It is worth noting that Pinstrup-Andersen and Franklin's (71). model is in a sense a formalized version of Castro and Schuh's analysis. Of course, it is formally much larger and more comprehensive, with subsystems for the demand side cast in terms of matrices of elasticities, etc. 4/ Brinegar (17) argues strongly in favor of taking account of indirect (second and higher order) effects of technological change through long term multipliers. Quoting another author in the area of education to the effect that "recent estimates of high economic return to education could be very misleading because the real returns are much higher," Brinegar implies that the same might be occurring with research because secondary effects are not followed through time. 5/ Bayley (13) in addition to warning against a "Greek philosopher's attitude" with respect to new methods for research resource allocation, calls attention to the need to better identify how benefits and losses are distributed. - 49 - 5.109 A point to note is that in Castro and Schuh's model no explicit treatment of the probabilities of success (technical, "commercial" or other) is presented. This is as difficult an aspect of ex ante models as it is important. The question involves how much difference it makes to discriminate even grossly among probabilities of success for alternative research endeavors relative to not discriminating at all. Not discriminating implies that for all practical purposes the probabilities are assumed to be equal. The question that arises is whether it is more desirable to give an equal probability of success to all alternative research projects than to try to discriminate among them. Ultimately, this is an empirical question. The answer may have to wait for further research. 5.110 Another point worth noting is that the Castro and Schuh model was developed within the framework of a given country, with given goals, institu- tions, policies and endowments. The international agricultural research system, even in the case of a Center concerned with only one commodity, operates in a much wider milieu. The task of following the distributional effects among all classes of exporters, importers, producers, consumers, regions, sectors, etc. becomes much greater. However, this problem is present whatever the allocation model chosen. The fact that some models dodge this obstacle by not giving any consideration to these aspects does not make them any more desirable. f. Easter and Norton Model 5.111 Easter and Norton (29) apply an ex ante benefit-cost analysis to requests for additional resources in the federal budget of the Land Grant Universities. Specifically, they consider the case of certain research program areas in corn and soybeans in the North Central region. They also discuss applications of benefit-cost analysis to livestock and to rural development research. 5.112 In discussing the criticisms that have been made against the use of benefit-cost ratios in ex ante analysis of research resource allocation, they argue that "while problems of estimating benefits preclude the determi- nation of an 'optimal' allocation of research resources, quantitative cost benefit techniques may help policy makers improve their decisions. Certainly as a minimum, carefully calculated estimates of benefits can be compared with costs to determine which projects will likely yield positive returns." 5.113 Benefit-cost ratios are calculated using the low side of a range ot estimates provided by scientists on the yield and cost effects of each research line, as well as on the expected adoption rates for the new tech- nology. Also, a discount rate is adopted (1OZ in the example) together with product prices and probabilities of success for each research alter- native. Some assumption is made about the trend in area cultivated with the crop, in the present example that it would remain constant at the base year (1975) level. Product quality was also assumed either unchanged or, if improving, not affecting the cost of livestock feeds. Benefits were followed through a period of 25 years ending in the year 2000. - 50 - 5.114 An important aspect of the analysis of required additional funds for research in the Easter and Norton approach is the sensitivity of benefit cost ratios to single and combined variations in the probabilities of success9 the expected yield increases, the product prices, and the length of the lag between research expenditures and the availability of the research results to the farmers. While it is easy to anticipate the direction of the changes in the benefit cost ratios due to changes in these conditions, the careful application of sensitivity analysis to any real, quantified evaluation, is a very important source of information to the decision-makers. As a matter of fact, it works as a kind of feedback process to let administrators know the relative importance of added precision and accuracy in the estimation of the several variables involved in the evaluation. 5.115 An effort was also made to assess the distribution effects of the research program. The increased-production effect of the research was spread over the related sectors of the economy. In the example used by Easter and Norton, the effect of a 3 percent increase in the production of corn and soybean on the feed, livestock, and meat economy was traced through. The price elasticity of demand for corn and for soybeans were used to assess the effect of increased production on farm gross income. These elasticities appear to be close to unity, since gross farm income is estimated to remain unchanged. Effects on the soybean oil market were also analyzed, as well as the long run effect on the livestock sector. 5-116 In evaluating research projects in livestock, the benefit-cost ratio, given the necessary adaptations, was still considered useful, in spite of a greater difficulty in assessing benefits. In the rural development research and extenslon project appraisal, Easter andW Norton turn to cost effectiveness as an operational method. 'fThey call for information on (1) the research and extension objectives, (2) a cost estimate by objective, and (3) expected outcomes in physical (in money if possible) terms. Alternative ways of reaching the same results are then assessed. 5-117 As one of their conclusions, Easter and Norton stress the key role that the cooperation between scientists and social scientists plays in the effort to evaluate ex ante research efforts. g. Atkinson-Bobis Model 5.118 Simulation, viewed in a broad perspective to include several random sampling procedures, has been utilized in several models for research resource allocation. In the studies reviewed above, simulation was normally used in the approaches of Fishel, Pinstrup-Andersen and Franklin, and Cartvright. 1/ Simulation is at the center of Atkinson-Bobis' model. 1/ In 1972D Souder (93) estimated that much more than one hundred models of research resource allocation had been built. Simulation procedures must have been more or less an important component of the solution approach to a good number of these models. - '1 - 5.119 The model developed by Atkinson and Bobis (8) for the industrial sector has fared well in the opinion of experienced research managers and of model builders.. 1/ It is not a pure simulation model since it involves optimization through dynamic programming. A random sampling procedure is used to take account of the stochastic nature of some of the variables involved in the procedure, producing a distribution of losses and of positive returns, in dollar terms, around the expected returns. 5.120 The Atkinson-Bobis model has a probability component relating expenditures and probability of successful completion of each project, in each year and after any number of years within the planning horizon (five years of investments and eleven years in total for the product life in their example). Probabilities of technical, legal, engineering, and commercial success are established, and the product of these probabilities, provided the project is completed, gives the overall probability of success. 5.121 Sales estimates for specified years are made and they are fitted into a logistic equation due to the observed fact that new product sales usually follow an S-shaped pattern. The sales function recognizes that the initiation of sales later than anticipated usually results in a smaller share of the market being gained after sales are stabilized. The penalty for starting sales later than in the first year after successful completion of the project is made severe in the Atkinson-Bobis model. 5.122 Some assumptions are made about selling prices in each year, as well as about manufacturing costs (profit margins may be used). Overhead and selling costs are assumed a constant proportion of sales revenue. 5.123 The revenue from sales in any given year, considering sales started in any previous (or current) year, is then computed. The present value of the revenue stream net of overhead, selling, and manufacturing costs provides an estimate of the net revenue for any given project assuming sales started in a given year. Weighting these values for differences in the starting year of sales with the probability of success of the project in each year gives an expected payoff. 5.124 An estimate of the expected discounted research expenditure is obtained by considering expenditures and probabilities in both cases of project failure and success in each year. Then, the expected payoff minus the expected expenditures provides a measure of the expected profit of the project. ' 5.125 The model also has a mechanism to analyze the rate of expenditure over time. Using concepts of efficient research expenditures, effective research expenditure, and the relationship between expenditures in one year and in the following years, an equation is derived to permit adjustment 1/ Souder (93) reports that the Atkinson and Bobis model came out well in a comparison of 26 models according to realism (weight - 4), flexibility (weight - 3), capability and facility of use (weight - 2, each), and cost (weight - 1). Shumway (88) calls attention to this comparison. - 52 - in efficient expenditures in each year. The mechanism penalizes "over- investment" in the beginning year of the project, due to the higher prob- ability of wasting resources. An optimization procedure is then utilized for the profit variable over a set of projects subject to a budget constraint. 5.126 Results from an example presented in Atkinson and Bobis indicated that resources should be concentrated in fewer projects than the usual research policy was funding. An iterative procedure was used to reduce the optimization solution by the number of years, going over it year by year and repeating the allocation process until only negligible improvement in profits was verified. 5.127 Atkinson-Bobis resort to a random sampling procedure since the optimization works with point estimates for all the variables involved and since an interval estimation is more secure and informative. Moreover, sensitivity analysis with the optimization procedure would be laborious and not recognize the different probabilities associated with the different values in the relevant range of some of the variables. This simulation through a random sampling methodology permits the generation of a distribution of results (both losses, In the negative range of profits, and positive returns) around the expected profit for each project (the expected profit unbiasedly estimated by the average profit of all the simulation runs for each project). 5.128 This is just one example of a model using a kind of simulation procedure. The model was designed to represent the conditions and environment of an industrial concern, and would have to be adjusted for applications in agriculture. However, the form of.the final output and some characteristics of the structure of the model make it a necessary consideration when evaluating alternative models. 5.129 The ex ante models described above are but a few examples of the large number of models that have been tried in actual applications. The degree of methodological sophistication ranges from the simpler scoring models to the more complex mathematical programming models. 5.130 The advantages of these models are that they provide a basis for decision-making with an eye to the future rather than on what has happened in the past, they pool inform,ation from a large number of qualified experts, and they provide a means of relating the research effort explicitly to a set of goals. The disadvantages are that those methods which draw on the opinion of a large number of experts can be quite costly and time-consuming, and that the pooling of a large number of opinions may do little more than pool ignorance. It is probably for these reasons that the more complicated methods have often been used only once, - 53 - 5.131 It is our judgment that one of the methods that seeks to obtain the informed judgment of a large group of experts may have some merit if they are used, say, once every five years. If used in this way, they can provide a source of ideas and suggestions, as well as informed judgment about the appropriateness of research strategies being used. On a continuing basis, we believe there is considerable merit in the approaches taken by Castro and Schuh and by Norton and Easter, since they provide a means of feeding some rigorous analytical research into the decision-making process. 5.132 An important need is for more development work with all the ^ ante models. In our view the D in R and D for these models has been slighted. VI. CONCLUSIONS AND RECOMMENDATIONS 6.1 The preceding chapters suggest that evaluating agricultural research is a complex task that presents the analyst with a wide range of alternatives and challenges. Methodological developments have probably outrun the avail- ability of data to use the analytical tools at hand. And the need for judgment on a wide range of issues suggests that the mechanical implementation of a given procedure or procedures could be more dangerous than productive. 6.2 Yet the need to evaluate agricultural research and its impact on the economy is clear. Budget procedures in most countries no longer permit the allocation of budget funds without some analysis of their alternative payoffs. Even more importantly, agricultural research is a clear case where private costs and returns diverge from social costs and returns. Hence, if resources are to be allocated to agriculturaL research in the socially optimal manner, some assessment of the costs and returns of such research is needed. 6.3 The tools discussed above also have other uses. They provide a means of understanding the contributions and social costs of agricultural research. With such knowledge in hand, policymakers can devise more rational science and technology policies. Similarly, a fuller understanding of the forces that technical change brings into play, and of the forces that in turn influence it, provides a guide to complementary policies that are needed for the research to make its maximum contribution to social and economic development. 6.4 Our principal conclusions and recommendations for research administrators policymakers, researchers, and research analysts at large, are as follows: a. The various methods, procedures, and approaches discussed above, should be used as a means and not as an end. The empirical and other results which they generate can be only one input into the larger decision-making process. They offer no panacea, nor should they be used as the sole basis for decision-making. b. An over-emphasis on evaluating research and assessing and monitoring research can stifle activity and destroy research entrepreneurship. Both of these are critical to having a vital research system. - 54 - c. Agricultural research can contribute to social and economic development in a number of important ways. To date, major emphasis has been on research that increases output directly, and relatively less has been placed on indirect contributions. The study and evaluation of research should have a broad perspective. d. A clear specification of goals is imperative if research is to be evaluated effectively or if research priorities are to be established. To the extent that the goals are in conflict, one with the other, some attempt has to be made to establish the trade-offs among the various goals. e. The goals for a given country or region will depend on the stage of development of the economic or political entity, and on the economic policies in effect. The same applies to indi- vidual research centers and to research systems. f. The identification and measurement of the side effects of agricultural research or technical change is essential if rational policies are to be devised. In some cases these side effects will represent net benefits; in other cases they may represent net costs. In general it will be difficult to compress the direct and indirect effects of technical change into a single-valued parameter. That is no excuse for ignoring the indirect effects. Their evaluation makes for a more precise measurement of the costs and benefits of research, and also provides the basis for determining what complementary policies are needed.- g. The various methods which attempt to evaluate agricultural research on the basis of historical data are best used in an attempt to understand the process of technical change and thereby to provide guidance for impr6ved policymaking. Clearly, the past can be a guide to the future, and in particular, a series of cost-benefit analyses and/or estimates of the social rates of return to investments in agricultural research can give the policymaker some notion of whether he has been under- or over-investing in agricultural research. But the future may be different from the past and so care should be exercised in extrapolating to an uncertain future. h. General equilibrium or "external" effects should be taken account of in estimating the costs and benefits of agricultural research. Under certain circumstances these effects will represent net costs. The important point is that there are some well-developed procedures for measuring such effects. i. The fact that cost-benefit ratios and the rate of return should be low does not mean that the researchers involved are not productive in the usual meaning of that term. The generation - 55 - of negative results has social benefit, even though with the present state of our knowledge we cannot measure it. Moreover, the social payoff may be low because of the economic policies being pursued. For these and other reasons, the benefit-cost framework and results should be used in drawing normative inferences with a great deal of care. j. The method chosen to evaluate research should be conditioned on the availability of appropriate data. Often, simple approaches which are less demanding in terms of data are more useful than more complicated procedures which have to be based on more precarious data. k. The various ex ante methods, or the methods which look to the future, are most appropriate in establishing research priorities. The advantage of these procedures is that they provide a formal means of using pooled judgment. 1. Methods which depend on the pooling of judgment should be complemented to the extent possible with hard data and evidence, and with studies which draw on historical experience. m. The use of scoring and other methods can be very demanding of highly qualified people. Hence, the systematic use of such procedures are recommended only when initiating new research programs and at about five-year intervals thereafter. n. Technological assessment should be an integral part of any significant research effort that is expected to have a significant effect on the economy. Technological assessment usually requires multi-disciplinary teams. The results from such assessments can usually be a valuable input into outreach- extension type programs, and to policymaking at large. O. The establishment of a sound science and technology policy is the key to developing a productive and efficient research effort. - 56 - References 1. Addison, L. E., J. W. Litchfield, and J. V. Hansen, "Managing Growth and Change in an R & D Organization: the Role of Dynamic Modelling," R & D Management, Vol. 6, No. 2, 1976, pp. 77-80. 2. Agriculture and Rural Development Department, "Technology and Agricul- tural Innovation: An Issues Paper," The World Bank, Washington, March 1977 (mimeographed). 3. Akino, M. and Y. Hayami, "Efficiency and Equity in Public Research: Rice Breeding in Japan's Economic Development," American Journal of Agricul- tural Economics, Vol. 57, No. 1, February 1975, pp. 1-10. 4. Allen, D. H. and T. F. N. Johnson, "Optimal Selection of a Research Project Portfolio Under Uncertainty," Chemical Engineering (London, No. 241, 1970. 5. Allen, J. A., "A Philosophy on Research Management," Paper presented to National Workshop on Research Management, Melbourne, November 1976. 6. Allen J. M., "A Survey into the R 6 D Evaluation and Control Procedures Currently Used in Industry," Journal of Industrial Economics, Vol. 18, No. 2, pp. 161-181. 7. Ardila, V. J., "Rentabilidad Social de las Inversiones en Investigacion d;e=j^roeAWC_oub1&,` MS-dissertation, JICA, Unikvera8dx4-Nactonal, Bogotia, 1973-. 8. Atkinson, A. C. and A. B. Bobis, "A Mathematical Basis for the Selection of Research Projects," IEEE Transactions on Engineering Management, EM-16, No. 1, February 1969, pp. 2-8. 9. Augood, D. R., "A Review of R & D Evaluation Methods," IEEE Transactions on Engineering Management, EM-20, No. 4, November 1973, pp. 114-120. 10. Ayer, H. W. and G. E. Schuh, "Social Rates of Return and Other Aspects of Agricultural Research: The Case of Cotton Research in Sao Paolo, Brazil," American Jorunal of Agricultural Economics, Vol. 54, No. 4 (Part I}, November 1972, pp. 557-569. 11. Baker, N. R., "R & D Project Selection Models: An Assessment," R & D Management, Vol. 5 (Special Issue on Project Selection), 1975, pp. 105-111. 12. Barletta, A. N., "Costs and Social Benefits of Agricultural Research in Mexico," Ph.D. Thesis, University of Chicago, Chicago, Illinois, 1971. - 57 - 13. Bayley, N. D., "Research Resource Allocation in the Department of Agri- culture," in Resource Allocation in Agricultural Research, edited by W. L. Fishel, University of Minnesota Press, Minneapolis, 1971. 14. Beatie, C. J. and R. D. Reader, "Quantitative Management in R & D," Chapman and Hall Ltd., London, 1971. 15. Bobis, A. H., T. F. Cooke, and J. H. Paden, "A Funds Allocation Method to Improve the Odds for Research Success," Research Management, Vol. 14, No. 2, (March 1971), pp. 34-49. 16. Bredahl, M. and W. Peterson, "The Productivity and Allocation of Research: U.S. Agricultural Experiment Station," American Journal of Agricultural Research, Vol. 58, No. 4 (Part I), pp. 684-692. 17. Brinegar, G. K., "Discussion: Evaluation and Planning of Research in the Experiment Station," American Journal of Agricultural Economics, Vol. 50, No. 5, December 1968, pp. 1161-1162. 18. Brown, B. and 0. Helmer, "Improving the Reliability of Estimates Obtained from a Consensus of Experts," Report P2986, The Rand Corporation, California, 1964. 19. Bush, G. P., "Bibliography on Research Administration, Annotated," Washington, D. C.: University Press, 1964. 20. Cartwright, R. W., "Research Management in a Department of Agricultural Economics," Ph.D. Dissertation, Department of Agricultural Economics, Purdue University, 1971. 21. Caty, G., G. Drilhon, G. Ferne, and S. Wald, "The Research System: Comparative Survey of the Organization and Financing of Fundamental Research," 3 Volumes, OECD, Paris, 1972. 22. Cetron, M. J. and J. N. Johnson, "Technological Forecasting in A Dynamic Environment," in Quantitative Decision Aiding Techniques for Research and Development Management, edited by M. J. Cetron, H. Davidson, A. H. Rubenstein, Gordon and Breach, Science Publishers, Ince., New York, 1972. 23. Cetron, M. J., J. Martino, and L. Roepeke, "The Selection of R & D Program Content--Survey of Quantitative Methods," in Quantitative Decision Aiding Techniques for Research and Development Management, edited by M. J. Cetron, H. Davidson, A. H. Rubenstein, Gordon and Breach, Science Publishers, Inc., New York, 1972. 24. Clarke, T. E., "Decision Making in Technologically Based Organizations: A Literature Survey of Present Practice," IEEE Transactions on Engineer- ing Management, EM-21, No. 1, February 1974, pp. 9-23. - 58 - 25. Dalkey, N. and 0. Helmer, "An Experimental Application of the Delphi Method to the Use of Experts," Management Science, Vol. 9, April 1963, pp. 458-467. 26. Dalrymple, D. G., "Global Agricultural Research Organization," World Food and Nutrition Study: The Potential Contribution of Research, Supporting Papers, National Academy of Sciences, July 1977, Vol. V, Study Team 14, Subgroup B (in press). 27. Dean, B. V., "Evaluating, Selection and Controlling R & D Projects," American Management Association, New York, Research Study 89, 1968. 28. Dean, B. V. and L. A. Roepcke, "Cost Effectiveness in R & D Resource Allocation," IRE Transactions on Engineering Management, Vol. EM-11, December 1972. 29. Easter, K. W. and G. Norton, "Estimates of Potential Returns from Added Research Budget fcr and Land Grant Universities," Staff Paper, Depart- ment of Agricultural and Applied economics, University of Minnesota, November 1976. 30. Economic Research Service, Technology Assessment: Proceedings of-an ERS Workshop, April 20-22, 1976, AGERS-31, US Department of Agricul- ture, Washington, D. C., 1977. 31. Evenson, R. E., "The Contribution of Agricultural Research to Production," Journal of Farm Economics, Vol. 49, 'December 1967-, pp. 1415-1425. 32. -9Vbnson, R.-E,. 'Gains and Losses from, AgricuItural Technology," University of the Philippines, November 1975 (mimeographed). 33. Evenson, R. E., P. M. Flores and Y. Hayami, "Costs and Returns to Rice Research," Resource Paper No. 11, Conference on Economic of New Rice Technology, IRRI, Los-Banos, Philippines, December 1976 (mimeographed). 34. Evenson, R. E. and Yoav Kislev,Agricultural Research and Productivity, New Haven and London, Yale University Press, 1975. 35. Falcon, Q. P., "The Green Revolution: Generations of Problems," Amexicaax Journal of Agxicultural Economics, Vol. 52, 1970, p_ 698-710. Information System and Experiment," In Resource Allocation in Agricultural Research, edited by W. L. Fishel, Minnesota University Press, Minneapolis, 1971. 37. Gear, A. E., "A Review of Some Recent Developments in Portfolio Modeling in Applied Research and Development," IEEE Transactions on Engineering Management, EM-21, No. 4, November 1974, pp. 119-125. - 59 - 38. Gear, A. E., "A Review of Some Recent Developments in Portfolio Modeling in Applied Research and Development, R & D Management, Vol. 5 (Special Issue on Project Selection), 1975, pp. 59-65. 39. Gear, A. E., A. G. Lockett and A. W. Pearson, "An Analysis of Some Portfolio Selection Models for Research and Development," IEEE Trans- actions in Engineering Management, Vol. EM-18, May 1971. 40. Gotsch, C. H., "The Green Revolution and Future Developments in Pakistan Agriculture," in Rural Development in Bangladesh and Pakistan (ed. by R. D. Stevens, et. al.), University press of Hawaii, Honolulu, 1976, p. 361). 41. Gray, J. A. and S. P. Wolff, "The Changing Face of Research and Develop- ment in the Public Sector, R. & D Management, Vol. 6, No. 3, pp. 105-108. 42. Griliches, Zvi, "Research Costs and Social Returns: Hybrid Corn and Related Innovations," Journal of Political Economy, Vol. 66, October 1958, pp. 419-431. 43. Griliches, Zvi, "Research Expenditures, Education, and the Aggregate Agricultural Production Function," American Economic Review, Vol.-54, December 1964, pp. 961-974. 44. Hampton, J. M., P. G. Moore and H. Thomas, "Subjective Probability and its Measurement," in Concepts and Applications of Modern Decision Models, MSU Business Studies, East Lansing, Michigan, 1976. 45. Hayami, Y. and R. W. Herdt, "The Impact of Technological Change in Subsistence Agriculture on Income Distribution," American Journal of Agricultural Economics, Vol. 59, No. 2, May 1977, pp. 245-256. 46. Hayami, Y. and W. Peterson, "Social Returns to Public Information Services: Statistical Reporting of U. S. Farm Commodities," American Economic Review, Vol. 62, No. 1, March 1972, pp. 119-130. 47. Hayami, Y. and V. W. Ruttan, Agricultural Development, An International Perspective, The Johns Hopkins Press, Baltimore, 1971. 48. Hertford, R. and A. Schmitz, "Measuring Economic Returns to Agricultural Research," in Resource Allocation and Productivity in National and International Agricultural Research, edited by T. A. Arndt, D. C. Dalrymple, and V. W. Ruttan, University of Minnesota Press, Minneapolis, 1977. 49. Hertford, R., J. Ardila, A. Rocha and C. Trujillo, "Productivity of Agricultural Research in Colombia," in Resource Allocation and Productivity in National and International Agricultural Research, edited by T. M. Arndt, D. G. Dalrymple and V. W. Ruttan, University of Minnesota Press, Minneapolis, 1971. 50. Johnston, B. F. and J. Cownie, "The Seed-Fertilizer Revolution and Labor Force Absorption," American Economic Revieu, September 1969. 51. Keeling, A, G.,."Comments on Dr. Simmonds" Paper, Costs and Benefits of an Agricultural Research Institute,"t R & D Management, Vol. 6, No. 1, pp. 43-44. 52. Kislev, Y., "The Economics of Agricultural Research, Some Recent Findings," Food Policy, Hay-1977, pp. 148-156. 53. Klein, K. K., "An Investigation into the Evaluation of Applied Research Projects in Agriculture," unpublished Ph.D. Dissertation, Purdue University, 1976. 54. Knight, F. H., "Risk, Unvertainty, and Profit," Houghton Mifflin Company, New York, 1921. 55. Latimer, R. G. and D. Paarlberg, "Geographic Distribution of Research Costs and Benefits," Journal of Farm Economics, Vol 47, May 1965, pp. 234-241. 56. Lindner, R. K., "Autonomy andAccountability in Planning Research: The Challenge for Research Management," Economics Department Paper No. 5, the University of Adelaide, South Australia, September 1976. 57. Lindner, R. K. and F. G. Jarret, "Measurement of the Level and Dis- tribution of Research Benefits," Discussion Paper, Economics Department, University of Adelaide, September 1975 (mimeograpa). 58. Loasby, B. J., "Choice, Complexity,.and Ignorance--An Enquiry Into Economic Theory.and teh Practice of decision-Making'," Cambridge University Press, 1976. 590 Luce, R. D. and H. Raiffa, "Games and Decisions--Introduction and Critical Survey,." John Wiley and Sons, Ince., 1967. 60. kahlstede, J. P., "Long-Range Planning at the Iowa Agricultural and Rome Economics Department Station," in Resource Allocation in A&ricultural Research, edited by W. L. Fishel, University of Minnesota Press, 1971, 61. Moore, R. L., "Methods of Determining Priorities in a Programe of Research," R 6 D Manatement, Vol. 5-(Special Issue on Project Selec- tion), pp. 59-65. 62. Moore, Jo R. and W. R0 Baker, "Computational Analysis of Scoring Models for R E D Project Selection," Management Science, Vol. 16, Mo. 4, December 1969. 63. Mottley, C. M. and R. D. 'euton,.o`The Selection Of Projects for Industrial Research," Operation Research, Volo 7p ,ovember-December 1959, pp. 740-751. - 61 - 64. National Research Council, World Food and Nutrition Study, Washington, D. C., National Academy of Sciences, 1977. 65. Nguyen, D., "Intersector-Distributional Implications of Agricultural Progress in an Open Economy: An Extension," American Journal of Agricultural Economics, May 1977, p. 374. 66. Nutt, A. B., "An Approach to Research and Development Effectiveness," IEEE Transactians on Engineering Management, Vol. EM-1O, September 1965. 67. Paulsen, A. and D. R. Kaldor, "Evaluation and Planning of Research in the Experiment Station," American Journal of Agricultural Economics, Vol. 50, 1968, pp.. 1149-1162. 68. Pearson, A. W. "Project Selection in an Organization Context," R & D Management, Vol. 5 (Special Issue on Project Selection), 1975, pp. 59-65. 69. .Peterson, W. L., "Return to Poultry Research in the United States," Journal of Farm Economics, Vol. 49, August 1967, pp. 656-669. 70. Pinstrup-Andersen, P. and F. Byrnes (eds.), Methods for Allocating Resources in Applied Agricultural Research in Latin America, Centro Internacional de Agricultura Tropical, Cali, 1975. 71. Pinstrup-Andersen, P. and D. Franklin, "A Systems Approach to Agricultural Research Resource Allocation in Developing Countries," in Resource Allocation and Productivity in National and International Agricultural Research, edited by T. M. Arndt, D. G. Dalrymple and V. W.Ruttan, University of Minnesota Press, Minneapolis, 1977. 72. Pinstrup-Andersen, P., N. R. de Londno and E. Hoover, "The Impact of Increasing Food Supply on Human Nutrition: Implications of Commodity Priorities in Agricultural Research and Policy," American Journal of Agricultural Economics, Vol. 58, No. 2, May 1976, pp. 131-142. 73. Puterbaugh, H. L., "An Appliation of PPB in the Agricultural Research Service," in Resource Allocation in Agricultural Research, edited by W. L. Fishel, University of Minnesota Press, Minneapolis, 1971. 74. Ramalho de Castro, J. P., "An Economic Model for Establishing Priorities for Agricultural Research and a Test for the Brazilian Economy," Ph.D. Dissertation, Department of Agricultural Economics, Purdue University, 1974. 75. Ramalho de Castro, J. P. and G. E. Schuh, "An Empirical Test of an Economic Model for Establishing Research Priorities: A Brazil Case Study," in Resource Allocation and Productivity in National and International Agricultural Research, edited by T. M. Arndt, D. G. Dalrymple and V. W. Ruttan, University of Minnesota Press, Minneapolis, 1977. - 62 - 76. Roumasset, J. A., "Rice and Risk--Decision Making Among Low-Income Farmers," North-Holland Publishing Company, 1976. 77. Rubenstein, A. H., "Economic Evaluation of Research and Development: A Brief Survey of Theory and Practice," Journal of Industrial Engineer- >Lng Vol. 17, 1966, pp. 615-620. 78. Rubenstein, A. H., "Research on Research: The State of the Arts in 19689," Research Management, Vol. 11, No. 5, September 1968, pp. 279-304. 790 luesell, E. J., "R & D Evaluation--The Total Program Approach," Financial Executive, London, No. 234, 1969. 80. Saunders, G. S., "Investment Decisions in Research and Development," Chemical Engineering, London, No. 234, 1969. 81. Schich, A., "The Road to PPB: The Stages of Budget Reform," in Resource Allocation in Agricultural Research, edited by W. L. Fishel, Universityof Minnesota Press, Minneapolis, 1971, 82. Schmitz, A. and G. Seckler, "Mechanical Agriculture and Social Welfare: The case of the Tomato Harvester," American Journal of Agri- cultural Economics,'Vol. 52, No. 4, November 1970, pp. 569-578o 83. Schuh, G. E., "OSome Economic Considerations for Establishing Priorities in Agricultural Research," presented at OLAC Meeting of Program Advisors in Agriculture, Mexico City, November 1972 (mimeograph). 84. Schuh, G. E., "The Exchange Rate and U.S. Agriculture," American Journal of Agricultural Economics, Vol. 56, 1974, pp. 1-13, 85. Schultz, T. H., The Economic Organization of Agriculture, New York, McGraw-Hill, 1953. 86. Schultz, T. W., "What Are We Doing to Research Entrepreneurship?" Agricultural Economics Paper No. 77:11, University of Chicago, March 1977 (Mimeograph). 87. Scobie, G. Di and Ro Posada T., The Impact of High-Yielding Rice Varieties in Latin America, With Special Emphasis on Colombia, Series JE-Ol, CIAT, Cali, 1977, 88e Shumway, C. R., "Allocation of Scarce Resources to Agricultural Research: Review of Methodology," American Journal of Agricultural Economics, Vol. 56, No. 4, November 1975, pp. 716-718. 89. Shumway, C. R., "Models and Methods Used to Allocate Resources in Agricultural Research: A Critical Review," in Resource Allocation and Productivity in National and International Agricultural Research, edited by To M. Arndt, D. G. Dalrymple and V. W. Ruttan, University of Minnesota Press, Minneapolis, 1977. - 63 - 90. Shumway, C. R. and J. D. Hwang, "Application of a Resource Allocation System in a Technology-based Public Organization, R & D 1Management, Vol. 6, No. 1, 1975, pp. 31-37. 91. Shumway, C. R. and R. J. McCracken, "Use of Scoring Models in Evaluating Research Programs," American Journal ofAgricultural Economics, Vol. 56, No. 4, November 1975, pp. 716-718. 92. Simmonds, N. W., "Costs and Benefits of an Agricultural Research Institute, R & D Management, Vol. 5, No. 1, pp. 23-28. 93. Souder, W. E., "A Scoring Methodology for Assessing the Suitability of Management Science Models," Management Science, Vol. 6, No. 2, pp. 81-86. 94. Steck, R,, "How Can Research on Research Contribute to a Better Manage- ment of University Research?", R & D Management, Vol. 6, No. 2, pp. 81-86. 95. Sukhatne, V. A., "The Utilization of High Yielding Rice and Wheat Varieties in India: An Economic Assessment," Unpublished Ph.D. dis- sertation, University of Chicago, 1976. 96. Technical Advisory Committee, CGIAR, "Priorities for International Support to Agricultural Research in Developing Countries," TAC Secretariat, Food and Agricultural Organization of the United Nations, Rome, 1976. 97. Theil, H., "On the Optimal Management of Research: A Mathematical Approach," presented at the Conference of the International Federation of Operations Research Societies, Oslo, Norway, July 1963. 98. Thompson, R. L., "The Metaproduction Function for Brazilian Agricul- ture: An Analysis of Productivity and Other Aspects of Agricultural Growth," Ph.D. dissertation, Department of Agricultural Economics, Purdue University, 1974. 99. Thompson, R. L. and G. E. Schuh, "Sources of Regional Differences in Productivity in Brazilian Agriculture," presented at conference on Growth, Productivity, and Equity Issues in Brazilian Agriculture, Ohio State Uniersity, January 13-15, 1975. 100. Tweeten, L. G. and F. K. Hines, "Contribution of Agricultural Productivity to National Economic Growth," Agricultural Science Review, Vol. III, No. 2, 1965, pp. 40-45. 101. U.S. Department of Agriculture, "Manual of Classification of Agricultural and Forestry Research," Science and Education Staff, USDA, Washington, D.D., June 1970. - 64 - 102. U.S. Department of Agriculture, "A National Program of Research for Agriculture,' Report of a Study Sponsored Jointly by the Association of State Universities and Land Grant Colleges and U.S. Department of Agriculture, U.S. Government Printing Office, Washington, D.C., October 1966. 103. Wallace, T. D. and D. M. Hoover, "Income Effects of Innovation: The Case of Labor in Agriculture,"Journal of Farm Economics, Vol. 48, No. 2, Hay 1966, pp. 325-336. 104. Welch, F., "Education in Production," Journal of Political Economy, Vol. 78, January-February, 1970, pp. 35-39. 105. Wennergren, E. B. and M. D. Whitaker, "Social Return to U.S. Technical Assistance in Bolivian Agriculture: The Case of Sheeps and Wheat," American Journal of Agricultural Economics, August 1977, pp. 565-569. 106. Williams, D. J., "A Study of a Decision Model for R & D Project Selection, Operations Research Quarterly, Vol. 20, No. 3, 1969. 107. Williamson, Jr., J. C., "The Joint Department of Agriculture and State Experiment Stations Study of Research Needs," in Resource Allocation in Agricultural Research, edited by W. L. Fishel, University of Minnesota Press, Minneapolis, 1971. 108. Wise, W. W., "The Role of CoseBenef it Analysis in Planning Agricul- tural R &D Programs," Research Policy, Vol. 4, pp. 246-261. - 65 - APPENDIX A The Iowa Model It is possible to represent the Iowa model algebraically in the following way: l/ Identify each project as Pi and let there be n projects to be evalu- ated, i.e., j - 1, 2, ..., n. Define C- I i iCij where C - cost of P in terms of C - input of resource i to P , and V - weighting scheme for the resource inputs (price of the re- source), and 01 - Pkgk, where C - growth contribution of P iln terms of 9k - contribution of P3 to growth aspect k (resources saved, increase in output). V - weighting scheme representing relative importance of each growth aspect considered, and E - Whe where K - contribution of P to the equity objective in terms of 1/ This presentation follows closely the one by Paulsen and Kaldor (67). 66 c contribution of Pi to equity aspect h (absolute and relative equity) V - weighting scheme representing the relative importance of each equity aspect considered, and S . x VWSri, where j r rr S - contribution of P to the security objective in terms of ar contribution of P to security aspect r (security of person, of property). Other aspects might be related to quality of pro- ducts, to consumer health, nutrition, life preservation, re- sources (soil) depletion, etc. I - weighting scheme representing the relative importance of each h security aspect considered, and BJ - W1Gi + W2*E + W3S , where Bj - contribution of P to overall goal of growth, equity, security-1 G1, IJ Si- as previously defined - ts t - 1,2,3, - weighting scheme representing the relative impor- tance of each goal (growth, equity, security) considered. Panel members as individuals and as a group have to provide estimates of j %. srj' define (supposedly social) preferences with respect to the relative importance represented by W9 W'h, and Wr, estimate C and define (aupposedly market) weights W., compute G%, ESD Si. and C for each PgD and finally compute growth-cost ratio G1/C1, equity-cost ratios /C i and security-cost ratios S1/CJ for each PJO To obtain a measure - Sometimes Bi is defined in a multiplicative instead of an additive way. See Ibttley and Newton (35). - 67 - of the contributiou of each Pi to the overall goal of growth-equity- security, the panels have to establish (social) preferences, Wt, t - 1, 2, 3 and compute Bi as previously defined. Then a B /C ratio can be computed, representing an index of total cost-effectiveness of each Pi. - 68 - APPENDIX B The Minnesota Model Let B and C stand for the (discounted) present value of benefits and costs, respectively. MARRAIS considers B and C stochastic variables that depend on the annual benefit from the research assumink 100 percent adop- tion of its findings (v), the adoption pattern over time (a(t)), on the "scrap" value of some research facilities (S), on the "process" value of research (Pr(t,c')). This relationship then is expressed algebraically as (1) b - fv t la(t)kt + Skt + Pv(t,c'), where T random value from the probability distribution (P(t)) of time re- quired to couplete the research activity; k- f comes from a probability distribution P(F) of technological feasibility of the project;. v comes randomly from a probability distribution P(V) of estimated annual average benefit at 100Z of adoption. S^imlarly, a single value c from the probability distribution C is assumed to depend on the average annual expenditure on the research acti- vity (C), on the mximum annual dissemination expenditure(E), and on the time path of dissemination cost of research (d(t)) starting one period after the project is completed (T + 1). The algebraic representation - 69 of this relationship is t tt (2) c - C' kt + E ! d(t)k t-1 t-T+l Note that the adoption pattern over time (a(t)), and the time path of dissemination costs (d(t)), are specified functions of time./ By specified group procedures2/ all the variables are estimated. "Estimators" are given alternative levels of CO for each project. Call it Ck, the level of average annual expenditure dn project j. Then, for each Cj, f"estimators" provide P(TJ ), a probability distribution of time to complete the research j funded at level k annually. Then, using the mean of P(Tjk) and Cjk, estimates of benefits are made, resulting in a pro- bability distribution P(B*j) of benefits from project J, funded at level k of annual expenditure, and given the mean expected time for completion. Now, Cjk, P(Tjk), P(B*k) and some specified level k of expenditure permit the computation of b and c through equations (1) and (2) above, for each pioject j. Note that the probability distribution of benefits estimated by the "estimators" P(B)k) is "weighted" by the probability distribution of technological feasibility P(Fjk) for each J, k-combination to give the resulting probability distribution P(Bjk) of benefits to be used to com- pute b./ A random sampling procedure from the probability distribution of the involved stochastic variables, single valued estimates of the other variables, and E(P(Tjk)), computations with equations (1) and (2), / The reader interested in their specific algebraic form may refer to Fishel (20). See Shumway (51) for a discussion of group procedures. A discussion of the problems of putting a value on the research results is found in Fishel (20). - 70 - the cost-benefit and ratio formulas, and the internal rate of return formulas, gives repeated, probability distributed estimates of these parameters, B-C, B/C, IRR. A large number of repetitions will increase the confidence of the estimate of B-C, B/C and IRR.- 1/ The estimates will converge to the parameters as the number of repetitions is increased. RECENT PAPERS IN THIS SERIES No. TITLE OF PAPER AUTHOR 332 Monitoring and Evaluation in the PIDER M. Cernea Rural Development Project - Mexico 333 Determinants of Private Industrial A. Pinell-Siles Investment in India 334 The 'Graduation" Issue in Trade Policy I. Frank Toward LDCs (consultant) 335 Balancing Trickle Down and Basic Needs M. Selowsky Strategies: Income Distribution Issues in Large Middle-Income Countries with Special Reference to Latin America 336 Labor Force, Employment and Labor Markets L. Squire in the Course of Economic Development 337 The Population of Thailand: Its Growth S. Cochrane and Welfare 338 Capital Market Imperfections and V.V. Bhatt, Economic Development A.R. Roe 339 Behaviour of Foodgrain Production J. Sarma and Consumption in India, 1960-77 S. Roy 340 Electric Power Pricing Policy M. Munasinghe 341 State Intervention in the Industrialization A. Choksi of Developing Countries: Selected Issues 342 Policies for Efficient and Equitable J. Linn Growth of Cities in Developing Countries 343 The Capital Goods Sector in LDCs- A Case J. Datta Mitra for State Intervention? 344 International Technology Transfer: Issues F. Stewart and Policy Options 345 Family Planning Programs: An Evaluation R. Cuca of Experience - 2 - No. Title of Paper Author 346 Prospects for traditional and Non- 1D Sughart Conventional Energy Sources in Developing Countries 347 National Urbanization Policies in R1 enaud Developing Countries 348 Private Direct Foreign Investment R. Billerbeck in Developing Countries 0 Yasugi 349 Adjustment Policies and Problems in N tNolf Developed Countries 350 Energy Options and Policy Issues in Do Pallen-Bailey Developing Countries T0 Byer 351 Growth and Equity in Semi-Industrialized J0 Bergsman Countries 352 Capital Flows and Developing Country Debt J0 Xats 353 Trade Policy for Developing Countries D. Keesing 354 Development Problems of Mineral-Exporting G. Nankani Countries 355 The Global Framework M. Cheetham S. Gupta A. Schwartz 356 The Distribution of Income in Brazil G. Pfefferman *R Webb 357 Estimating Shadow Prices for Colombia W. SchohL in an Input-Output Table Framework 358 Inter-Country Comparison of 'Realw (PPP) P0 Isenman Incomes: Revised Estimates and Unresolved Questions 359 Price Distortions in Agriculture and Their E. Lutz Effects: An International Comparison M. Bale