5.,        ni§S; \- -; w : -24917
;w-s >s /   sW  \ s \ zi2002
*  X    ,--  [E  t 2-  DS-  },;R  ,            ...-- 7.
A~~~ K               ~. . . \ ;¢u
Sr ~~_ _U                               5
r5                 ' '  _  _* e : . >   M I t   s K : .   >  7 i's' : :,&-2_!2t'2Jl ; e..
* :-   .e                                 i ..rs '. ''    ,,, '   s £~ ' ~ ~   '
IFF~~~~~~~~~~~~~~~~~~~~~~~~~~~I
^-s'                             iti''''  s ''"
b'                             t]           , ;' ' ',:'''''~ 
; ^  -     ,,,, ,1  a' t^|@|Ft5t  ^''*rJ'~~~~~~~~~~~~~~~~.5ts 
,,   , <   S   .  6   ,  ffi  .   .   A  <   %  ,  .   , ffi   3   i
k ~ ~ ~ ~ ~ ~ S ,        '0 1   -;,-  ; ,



THE WORLD BANK OPERATIONS EVALUATION DEPARTMENT 
I~~~~~~~~I
OED: The First 30 Years
A Chrestomathy
2002
The World Bank
Washington, D.C.



THE WORLD BANK OPERATIONS EVALUATION DEPARTMENT
OED: The First 30 Years
A Chrestomathy
2002
The World Bank
Washington, D.C.



OPERATIONS EVALUATION DEPARTMENT
ENHA4NCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE
AND INDEPENDENCE IN EVAL UATION
The Operations Evaluation Department (OED) is an independent unit within the World Bank; it
reports directly to the Bank 's Board of Executive Directors. OED assesses what works, and what
does not; how a borrower plans to run and maintain a project; and the lasting contribution of the
Bank to a countrv 's overall development. The goals of evaluation are to learn from experience, to
provide an objective basis for assessing the results of the Bank's work, and to provide accountability
in the achievement of its objectives. It also improves Bank work by identifving and disseminating the
lessons learnedfrom experience and by framing recommendations drawn from evaluation findings.
OED Working Papers are an informal series to disseminate the findings of work in progress to
encourage the exchange of ideas about development effectiveness through evaluation
The findings, interpretations, and conclusions expressed here are those of the author(s) and do not
necessarily reflect the views of the Board of Executive Directors of the World Bank or the
governments they represent.
The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries,
colors, denominations, and other information shown on any map in this work do not imply on the part
of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance
of such boundaries.
Contact:
Operations Evaluation Department
Partnerships & Knowledge Programs (OEDPK)
email: ecampbellpage@worldbank.org
e-mail: eline@worldbank.org
Telephone: 202-458-4497
Facsimile: 202-522-3125
http:/www.worldbank.org/oed



OED: The First 30 Years
A Chrestomathy
The fiscal year 2003 (beginning July 2002) marks the 30th year of the
establishment of the Operations Evaluation Department (OED) at the
World Bank. This note briefly traces-through a select "chrestomathy"*-
the evolution of the evaluation function at the Bank, some of the emergent
challenges, and the institution-wide response to these challenges.
The 1970s
The Beginnings
I feel that, with the growth of the Bank Group, the time has come
to carry our systematic auditing procedures a stage further. I have
therefore decided to establish a unit whose sole responsibility will
be to evaluate the contribution of our operations to the
development of member countries. The principal task of this unit
will be to review past lending operations with the central objective
of establishing whether the actual benefits of the completed
projects are in accordance with those expected at the time of
appraisal and, in case of divergence, the reasons. I believe that this
initiative will have two principal benefits. It will contribute to the
formulation of policy and procedures by enabling us to learn more
from our past experience. And it will give us a degree of conviction
about the impact of what we are doing which we could obtain in
no other way.
Administrative Circular: Establishment of on
Operations Evaluation Unit, signed by Robert S.
McNamara, President, September 2, 1970.
* The word "chrestomathy' was formed and first used in Greek, from the words khrestos, useful, and mathein, to
know, hence useful learning. Its most frequent sense these days is that of a selection of passages designed to
help in learning a language. The most popular title is that of A Mencken Chrestomathy, a selection of the choicest
writings of the American journalist and writer on language, the late H.L. Mencken. Source: Michael Quinion,
World Wide Words. 1996-.
1



A distinction may usefully be drawn between "Audit' and
"Evaluation." In the work of the Operations Evaluation Division
'Audit" has come to mean comparison between the targets and
projections contained in the project appraisal reports on the one
hand and actual developments on the other, in order to see
whether or not appraisal objectives were attained; if not, why not
and if so, was it due to the project? "Evaluation," on the other
hand, has come to mean a deeper analysis of the contribution to
development made by the project and by the Bank in its support
of the project, with a view to seeing not only whether the project
met its assigned objectives but also whether these objectives were
themselves correct in retrospect and whether they might have
been improved in any way.
Memorandum by the President to the
Executive Directors: Future Work Program in
Operations Evaluation, July 20, 1972.
Independence of the Evaluation Function
The unit responsible for this work has always had a high degree of
independence from management, and it has grown in stature and
experience its independence has been increasingly formalized
most recently by making it a separate department in July 1973
under a Vice President without other direct operating
responsibilities and giving it a formal link with the Joint Audit
Committee of the Executive Directors in November 1973.
Report of the Joint Audit Committee for 1973,
October 25, 1973 (Document R73-243/1); and Terms of
Reference of Joint Audit Committee, November 21, 1973
(Document SecM73-704).
I think that the time has now come for the growing link between
the Executive Directors and the Operations Evaluation Unit to be
recognized formally by drawing lines on our Organization Chart
directly linking the unit, and the Vice President in charge of it, to
the Executive Directors as well as to the President. I also propose
that, following the retirement in 1975 of the Vice President
presently responsible, and in any case not later than December 31,
1975, a title of Director-General, more indicative of independence
from the normal operations of the Bank, should be instituted for
the manager of this function. Appointees would hold office for
2



renewable terms of five years, be removable only by the Executive
Directors and be ineligible for subsequent appointment or
reappointment to the Bank Group staff except in unusual
circumstances.
The basic work program of the unit would be reviewed and
approved periodically by the Executive Directors and all reports of
the Operations Evaluation Unit would be sent simultaneously to
the Executive Directors and to the President, without any
interference by the latter with their content.
Memorandum by the President to the Executive
Directors: Operations Evaluation, September 27, 1974.
The Evolving Operations Evaluation System
Project Performance Audits will be prepared on all projects in
developing countries about one year after completion of
loan/credit disbursement (i.e., typically 4 to 8 years after approval
by the Board of Directors). The feasible sophistication of this
enquiry, one year after completion of Bank/IDA disbursements,
will vary considerably among projects, depending on the nature of
the objectives sought, the sector to which the project belongs and
the duration of disbursements.
Memorandum by the President to the Executive
Directors: Operations Evaluation and
Project Performance Audits, April 13, 1973.
This report is a summary review of the first 50 projects covered by
Project Performance Audit Reports which were issued between
October 1972 and March 1975. The overall average economic rate
of return to the $3,200 million of investment is now estimated at
about 15%, and this figure is not very dependent on the changes in
relative prices that have occurred in the last few years. This is
considered a good performance on the part of the Bank as a
development lending institution, especially in view of the difficult
and changing environments, natural,,political and economic, in
which many of the projects were implemented.
First Annual Review of Project Performance Audit
Results, September 26, 1975.
3



In accordance with the basic principle that, for most effective
learning and impact on future operations, evaluation should be
carried out by those directly involved in the operational side of the
Bank's work, most of the evaluation activities are of a
decentralized and self-evaluative nature.
Operations Evaluation, World Bank
Standards and Procedures, June 1976.
It has long been Bank practice to require progress reports from its
borrowers. This is a regular part of the supervision process. In the
past, these reports have focused primarily on the typically clearly
specified and monitorable activities for which project monies are
being disbursed (such as the construction of a road, an irrigation
facility, a power plant or a school, or the commitment of monies by
financial intermediaries to sub-borrowers), and related institution
building. They have not focused as consistently on the impact of
the projects on the society and on the intended beneficiaries;
especially where, as in the fields of agriculture and rural
development, education, population and urban development,
special efforts are needed to establish a measure of the situation
the project is intended to alter if the changes effected by the
project are to be observed.
Operations Evaluation-
Annual Report, October 13, 1976.
The 1980s
From Projects to Policies
Policy-based lending is appropriate in the right circumstances,
whether it is called program lending, as it was called in the 1960s,
or structural adjustment lending, as it is called in the 1980s. But my
own view is that policy-based lending is not the answer to all
development problems; there is a role for intelligently conceived
and well designed investment project lending. But most important
is the role of intelligently conceived and managed country lending
4



programs. These lending programs are for me the key operational
instrument of the Bank; that is, the aggregation of the structural
adjustment loans and the sector loans and the project loans which,
all together, can be designed to focus on an agenda of issues with
the countries. The real Bank lever, as I see it, is neither an SAL nor
a project, but the set of activities called the country program.
Transcript of interview with Mervyn L. Weiner
(first DGO, 1975-84), July 1986.
One of the most important services the Bank can provide is to
ensure that the process of policy reform is "internalized" in the
country as quickly as possible, so that the reform program is
designed by the country itself and integrated into its long-term
development program.
"Lessons Learned and Implications," in OED,
Structural Adjustment Lending: A First Review
of Experience, September 24, 1986.
Structural Adjustment Loans (SALs) are rather difficult to evaluate
because structural adjustment involves the interaction of many
variables about which we still have much to learn. The first SALs
were implemented in a very difficult world economic
environment. SALs were a rapidly evolving product and their
evaluation was like focusing on a moving target.
'All about OED," Interview with Yves Rovani,
The Bank's World, December 1986.
Bank-Counhty Relations (and Special) Studies
The Bank's successful mediation efforts in the 1960 Indus Waters
Treaty between India and Pakistan has awarded the Bank the
benefit of doubt ever since.
[In Pakistan], the boundary line between the givers and receivers
of aid became a fluid one. When necessary, the Planning
Commission and the Finance Ministry could seek the alliance of
the [Aid to Pakistan) consortium in persuading other departments
5



to accept a change in policy. On other occasions, the World Bank,
the USA, or some other members of the consortium could seek the
alliance of the Government of Pakistan in order to impress its own
view on the consortium as a whole.
OED, World Bank in Pakistan: Review
of a Relationship, 1986.
OED in the Reorganized Bank
The mandate and the independence of the Operations Evaluation
staff have been reaffirmed by the reorganization. Internally, our
independence helps ensure the choice of subjects, access to
information and candor of reporting necessary for accessing the
accountability and for drawing fully on the benefits of experience.
Externally, it contributes to the credibility of the Bank's
achievements by demonstrating publicly its openness and
willingness to be the first to recognize its own shortcomings. The
increased challenge is to revitalize: evaluation, the process by
which we account for results and draw lessons from experience;
dissemination, the way we share the experience across the Bank;
and feedback, by which experience is reflected into new
operations.
Yves Rovani, "The Bank's World:
An Editorial," January 1988.
The 1990s
Ensuring Quality at Project Entry
To improve upon the Bank's portfolio performance management,
the Task Force advances six principal recommendations and a
comprehensive program of measures for their implementation: (a)
introduce the concept of country portfolio performance
management linked to the Bank's core business practices; (b)
provide for country portfolio restructuring in adjusting countries
including the reallocation of undisbursed balances of loans/credits;
(c) improve the quality of projects entering the portfolio through
better project analysis, appraisal methodology, and contractual
arrangements; (d) define the Bank's role in, and improve its
practices of project performance management; (e) enhance OED's
6



role as an instrument of independent accountability and refocus ex
post evaluation on sustainable development impact; and create an
internal environment supportive of better portfolio performance
management.
Effective Implementation: Key to Development Impact,
Report of the Portfolio Management 'ask Force (the
"Wapenhans Report"), September 22, 1992.
Shifts in Evaluation Criteria
Conceptually, the Bank's evaluation criteria have evolved to reflect
the shifts in development paradigms over the decades. Gradually,
the economic evaluation of projects became far more
comprehensive and multidisciplinary than before. The main "unit
of account" in the Bank's operational work shifted from the project
to the country. The full mix of lending and non-lending services
that make up country assistance programs has become the
dominant preoccupation of development policymakers.
Sustainability of projects and the institutional development impact
of projects began to be systematically rated beginning in the early
1990s. Priority attention began to be been given to the relevance (in
addition to efficacy and efficiency) of project goals reflecting
country assistance priorities.'
Robert Picciotto, "Evaluation in the World Bank:
Antecedents, Methods, and Instruments," from The
International Atlas of Evaluation.
Real-Time Evaluations
Nearly fifty years of experience indicates that the vast majority of
the Bank's operations do in fact comply with our own policies and
procedures. The Inspection Panel provides a safety net in
exceptional cases where those high standards are not met-and in
that sense, offers a link between the Bank and the people affected
by the operations we help finance.
World Bank News Release: "World Bank
Establishes Independent
Inspection Panel," September 23, 1993.
1. An overall development effectiveness index has recently been introduced to reflect the combined
consideration of outcomes, sustainability. and institutional development impact.
7



The Bank's Strategic Compact established a 'stretch goal" of zero
defects in Bank workmanship. To that end, [Quality Assurance
Group] QAG was established in 1996 with a mandate to increase
accountability by conducting real-time assessments of the quality
of the Bank's performance in its major product lines. An additional
objective of QAG is to enhance learning and identify best practice,
so as to catalyze changes in policies, programs and procedures.
QAG Web site.
Partnerships and Evaluation Capacity Development
Poverty reduction was the theme of the third Biennial World Bank
Conference on Evaluation and Development, held in June 1999.
The conference was sponsored jointly by two Word Bank
departments, OED and the World Bank Institute. It brought
together more than 250 representatives of the international
development evaluation community, including bilateral and
multilateral donor agencies and civil society. Conference
participants discussed lessons learned from evaluations of poverty
reduction programs; ways to advance the methodological frontier;
partnerships, ownership, and participation in evaluation. The
participants also explored challenges for the evaluation of poverty
reduction programs, both in methodology and in setting
substantive priorities.
Evaluation and Poverty Reduction, World Bank
Series on Evaluation and Development (Volume 3).
The main precondition for developing a national evaluation
system is country demand.... There are particular risks if the
impetus for an evaluation system is donor-driven.... Unless and
until countries reach their own conclusions about the merits of
evaluation, an evaluation system is unlikely to be sustainable.
Public Sector Performance-The Critical Role of
Evaluation, Selected Proceedings from
a World Bank seminar, 1998.
8



The 2000s
Toward a Comprehensive Development
Agenda, and an Increasingly Complex Mandate
As it seeks to implement an increasingly crowded agenda, the
Bank must learn to reconcile "client ownership" of activities with
the Bank's corporate priorities; to adapt global knowledge to local
conditions; to balance support for countries with strong enabling
environments with support to countries with poor enabling
environments but a high incidence of poverty; and to be efficient
and selective in implementing a holistic vision of development,
making judicious use of partnerships.
Annual Review of Development Effectiveness, 2000.
The Bank has also taken upon itself the goals of generating and
disseminating development knowledge, engaging its partners in
global public policy debates, and dealing with global development
challenges that cannot be tackled effectively at the country level.
These institutional responses have given rise to tensions and trade-
offs among goals.
Today's interview with Robert Picciotto,
February 22, 2001.
Sharing Knowledge on Development Evaluation
The impetus for the program originates in a growing global
demand for professional evaluation of development policies,
strategies, programs, and projects. A rapidly changing
environment, increasing public pressures for accountability and
results, as well as an emerging consensus behind more
comprehensive development strategies underlie the keen interest
in acquiring up-to-date evaluation knowledge and skills within the
development community.2
Today's interview with Ray Rist (Senior
Evaluation Officer), July 23, 2001.
2. July 2001 saw the launch of an annual summer program for executive training in development evaluation, the
International Training Program in Development Evaluation Training (IPDET)-an effort of OED in collaboration
with Carleton University in Ottawa. Approximately 130 participants from 41 countries attended the first offering
of this annual event.
9



We the participants of the Preparatory Meeting on IDEAS that was
jointly sponsored by UNDP Evaluation Office and the World Bank
Operations Evaluation Department, representing evaluators and
development practitioners, mostly from the developing world,
declare their commitment and support towards the establishment
of an International Development Association (IDEAS). We believe
that IDEAS would serve as an effective platform advocating
development evaluation as a key tool for managing for results,
promoting transparency and accountability in decision-making
and governance processes, and facilitating knowledge learning on
evaluation and development at the global level.
London Declaration on the
Establishment of IDEAS, 7-8 May 2001.
10