W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T OEDREACH FEBRUARY 24, 2003 Independence of OED The extent to which an evaluation unit is independent is a perennial issue in evaluation oversight. A number of official evaluation and audit organizations have addressed this issue and compiled a set of criteria they suggest be used to define the degree of independence of evaluation and audit units. OED has applied the criteria recommended by these organizations to its own situation to assess its independence. This is a summary of the self-evaluation. The conclusion is that OED exhibits a high degree of independence, meeting or exceeding all criteria for independence. There are four major criteria of independence: · Organizational independence · Behavioral independence · Avoidance of conflicts of interest · Protection from external influence Organizational independence. OED's organizational independence is clear. The DGO reports to the Board of Executive Directors, not to Bank management, and appoints the Director of OED, through whom the DGO directs the work of all OED staff. OED reports its findings to the Board without Bank management's pre-clearance. Behavioral independence. OED has a long track record of critiquing Bank work, recognizing the Bank's achievements but also addressing its shortcomings and making recommendations for improvements. In recent years, for example, OED has used its reports to urge the Bank to adopt a results-based management approach, even though Bank management was not yet ready to do so. A long-time external observer of OED has told us that this "willingness by OED speak out is hardly new; rather, it has been a salient characteristic of the unit since its inception." Protection from external influence. OED is protected from outside interference in many ways. Its work program and budget are decided in concert with the Board, not Bank management. It has unrestricted access to all Bank records and staff in conducting its work, and can consult with government officials, the private sector, NGOs, media, and all sources it deems appropriate without prior clearance from Bank management. It controls the hiring, promotion, and firing of its own staff within the framework of the Bank's merit-based personnel system. One issue, however, is the clearance role of the Bank's networks in hiring and promotion decisions. Avoidance of conflicts of interest. OED has procedures in place, based on the Bank's Staff Manual, to protect against a wide range of conflicts of interest that could weaken the objectivity and integrity of OED's evaluations. These include protections against official, professional, financial, and familial conflicts. Independence of OED--Detailed application of criteria The following applies to OED four criteria of independence derived from evaluation and audit organizations listed at the end of this note. Criteria are shown in bold, specific indicators in italics, and commentary in regular text. Criterion: Organizational independence Direct line of reporting relationships between the unit and the Management and/or Board of the institution The DGO's mandate makes the DGO "directly responsible to the Board of Executive Directors" for all functions under his or her control. The DGO is appointed by the Board to a fixed term, is removable only by the Board, and is ineligible for other positions within the Bank at the term's end. The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated The Director of OED is appointed by the DGO in consultation with the Chair of the Board's Committee on Development Effectiveness (CODE) and the President. The Director reports to the DGO and in turn appoints and supervises the work of OED staff. Thus, OED staff is organizationally outside the line of Bank management. The unit reports regularly to the larger organization's audit committee or other oversight body OED reports its evaluation findings to the Board through the DGO, including individual evaluations and broader syntheses of evaluation results. The unit is sufficiently removed from political pressures to be able to report out findings without fear of repercussions The World Bank's Staff Manual Principle 3 prohibits staff, including those in OED, from accepting instructions, remuneration, benefits, favors, or gifts of significant value from any government, or other external organization or person. This rule is backed by disciplinary procedures. Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit OED staff is covered by the Bank Group's personnel system, which is based on merit principles covering-- inter alia--compensation, training, tenure, and advancement. Criterion: Behavioral Independence Ability and willingness to issue strong, uncompromising reports OED reports routinely evaluate and critique the Bank's work at the project, country, sector, and global levels. While recognizing the Bank's achievements, these reports also address its shortcomings, draw lessons from its experience, and make recommendations to advance its development work beyond current practice. Transparency in the reporting of evaluation findings OED reports have always been available to internal stakeholders. Many reports also have been made available to external stakeholders, including nearly all Country Assistance Evaluations and Sector and Thematic Evaluations. Changes in the Bank's Disclosure Policy, adopted by the Board in 2000 and effective in 2001, make virtually all OED reports available to the public. Criterion: Avoidance of conflicts of interest Extent to which there are policies and procedures in place to identify and ameliorate any impairment from evaluator relationships; these policies and procedures are communicated to staff through training and other means; and they are enforced Conflicts of interest are addressed under Staff Manual Principle 3, which provides for disciplinary measures in case of violation. Staff is advised of these rules through Bank-wide training. OED managers are responsible for identifying and correcting any conflicts of interest. Extent to which stakeholders are consulted as part of the evaluation process to ensure against evaluator bias OED procedures allow stakeholders to have the opportunity to comment on OED evaluations at the stage of design, background studies, and before a draft report is finalized. Where appropriate, OED employs participatory evaluation methods to engage stakeholders, including beneficiaries, in the evaluation process. Extent to which rules or staffing procedures that prevent staff from evaluating programs, activities or entities for which they have or had decision-making or financial management roles, or with which they are seeking employment, are present and enforced OED staffing rules prohibit assigning evaluators to studies of programs, activities, or entities for which they have had prior decision-making or financial management roles. Staff members are required to disclose to OED management if they are seeking employment with entities they are evaluating during the course of the evaluation so that ameliorative actions can be taken, if necessary. The Bank's staffing manual includes specific limitations on future external employment in connection with official Bank duties, backed by disciplinary procedures. These rules are monitored and enforced by OED management. Extent to which rules or staffing procedures are in place to prevent staff from evaluating programs, activities or entities in which they have a financial interest are present and enforced Staff Principle 3 (applicable to all Bank staff) would generally prohibit OED staff from engaging in evaluations of programs, activities or entities in which they have a financial interest. Failure to disclose such an interest is subject to discipline. OED management enforces this rule. Extent to which rules or staffing procedures are in place to prevent staff from evaluating programs, activities or entities in which family members have influence are present and enforced Such conflicts of interest are addressed under Staff Manual Principle 3, which provides for disciplinary measures in case of violation. Staff is advised of these rules through Bank-wide training. OED managers are responsible for identifying any correcting such conflicts of interest. Criterion: Protection from outside interference Extent to which the evaluation unit is able to determine the design, scope, timing, and conduct of evaluations without Management interference The Board, not management, reviews OED's work program. The DGO and the Director (and not Bank management) oversee the design, scope, timing, and conduct of evaluations. OED has unrestricted access to all Bank records and staff in conducting its evaluations. Beyond that, OED can consult with all principals (including government officials, the private sector, media, NGOs, and other sources) without prior clearance from operational management. Extent to which the evaluation unit has control over staff hiring, promotion, and firing, within a merit system OED staff is covered by the Bank's merit-based personnel system. Within the framework of that system, OED management has the ability to hire, promote, and fire OED staff. An issue is the clearance role of Bank-wide sector boards in hiring and promotion. Extent to which the evaluator's continued employment is not jeopardized for reasons other than competency or the need for evaluator services The Bank's merit-based personnel system protects OED staff from firing, except for cause. Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities OED's budget is submitted to and approved by the Board, and is not subject to pre-review by Management. Extent to which the evaluator's judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority OED management, under the direction of the DGO, makes all decisions on the content of an OED evaluation report. Sources of criteria: · Canadian International Development Agency, CIDA Evaluation Guide (2000); · Danish Ministry of Foreign Affairs, Evaluation Guidelines (1999); · European Federation of Accountants, The Conceptual Approach to Protecting Auditor Independence (2001); · Institute of Internal Auditors, Professional Practices Framework (2000); INTOSAI, Code of Ethics and Auditing Standards (2001); · OECD/DAC, Principles for Evaluation of Development Assistance (1991); · U.S. General Accounting Office, Government Auditing Standards, Amendment 3 (2002). Task Manager: Patrick Grasso (202) 473-2568 Group Manager, OEDPK: Osvaldo Feinstein (202) 458-0505 · Director (Acting): Nils Fostvedt (202) 458-20719 OED Reach available @ http://worldbank.org/oed