T VJ1 CiICF,LS.-".IB~ I PEFAr1) 1j,l E iALUTCi J?FJIET WP% ~~~~~~21094 April 1999 I Evaluations DCD IIQB(ifi; Pc<,r J&r1s's--'1 WOWRLLBD BANKK OPPREARAT IOONNES EVALLUATUIOION D EDPART MAETNMT _ED Case Study Evaluations OED Working Paper Series * No. 2 Linda G. Morra Amy C. Friedlander Spring 1999 The World Bank r www.worldbank.org/html/oed Washington, D.C. Case Study Evaluations Linda G. Morra Amy C. Friedlander Overview In OED, much use is made of the case study. For example, the audits of two water projects in Malawi (World Bank 1997c)' and five transportation projects in Thailand (World Bank 1997d) and our study of paddy irrigation and water management in Southeast Asia (Rice 1997) are essentially all case studies. We use case studies for in-depth consideration of the results of a project or group of projects or to illustrate given points. Case studies are convincing and capture the reader's attention. But they are not generalizable; a case-no matter how well done-cannot tell you whether it is the only such instance or whether the problem (or success) is widespread. There is considerable confusion about what is and is not a case study, what is case study methodology, what use can be made of them, and how they should be written-up for dissemination. This paper attempts to clarify these issues. Definition of a Case Study Explanatory. The purpose of explanatory case studies, as the name implies, is to explain the relationships among program A case study is a methodfor learning about a complex components. instance, based on a comprehensive understanding of 1. Program implementation. This case study investigates that instance obtained through extensive description operations, often at several sites, and often normatively. and analysis of that instance taken as a whole and in 2. Program effects. This case study examines causality and its con text. usually involves multisitc and multimethod assessments. Site visits are generally associated with case studies, but not all Descriptive. These studies have a narrower focus than the site visits are case studies. We may do site visits to villages in- explanatory cases. country to observe or to utilize other data collection methods, 3. Illustrative. This type of case study is descriptive in character such as interviews. These site visits, however, do not necessarily and is intended to add realism and in-depth examples to use case study methodology. The case study method involves the other information about a program, project, or policy. elements of the definition presented above. That is, it entails 4. Exploratory. This is also a descriptive case study, but is aimed at generating hypotheses for later investigation rather than comprehenisive understanding and exteiisi've description and analysis of the instance as a whole and in its context. being illustrative. 5. Critical instance. This examines a single instance of unique Types of Case Studies interest or serves as a critical test of an assertion about a program, project, problem, or strategy. There are three main categories of case studies-explanatory, Combined Methodology descriptive, and combined methodology. While in real life there 6. Cumulative. This brings together findings from many case is often overlap among the types, one of the following ap- studies to answer an evaluation question, whether descrip- proaches will predominate. tive, normative, or cause-and-effect. 3 Size of the Case Study An event (Rwanda emergency relief effort) * A region, nation, or organization (Northwest Brazil, South We tend to think of a case study as small, but the size of the Asia and Pacific Region, India, UNESCO) instance can varv greatly. Sometimes the case is larger rather * "Nested" units in a large or complex case study (Tunisia, than smaller. For example, a case may be one site within a first through the seventh water supply project, or Malawi, project, three sites within a project, one project, a group of Country Assistance Review). projects, all projects within a sector in a given country, all projects within a sector across a set of countries, or all projects The Korea impact evaluation is an example of a case study that within a country. A country may be a case. The question is, what focuses on a region (see box 1). is the "instance that we want to take as a whole"? The instance as a whole can be: It is important to note that because an evaluation covers one or a small number of instances, this does not necessarily make it a * A site (fishery at Songkla Lake in Thailand) case study. An important part of the definition of case studies is * A function (public sector management) the phrase: "obtained by extensive description" If minimal * A project (Daxinganling, China Forest Fire Rehabilitation information was collected on a site with little depth of inquiry, it Project; Indonesia, University Development Project; or would not appropriately be classified as a case study Field work Equatorial Guinea, Technical Assistance Project) or site visits are not in themselves case studies. * A policy (promoting gender equality) * An office or department (Department of Education) The Kingdom of Morocco impact evaluation is an example of a case study that focuses on a region (see box 2). 4 Box 2.: Kingdom of Morocco: Socioeconomic Influence of Rural Roads, Fourth Highway Project The study,conducted during995,aims to understand the impact ofrural roads,fiveto tei yiears after completion of the improvements carried out undertheproject. Itfocusedon the impacts ontran3sport infrastructure andservicesagiculture,social servIceand theenvironment. It also assessestheeconomicbenefitsoftheimprovementsandtheirsustainability. Thestuy min findingistai thebenefits of paving rural roadsextendconsiderablybeyondaingtraffic easier; loweringthe cost of operat-g cars,trucks, andthervehicles;andimprovingthe qualityoftransport semices.The extendedbenefits includetriggeringmajorchanges in the agricultural economy, induding higher output; transformation of the agricultuMloutputnmix,ex pe fmlow-valuecerealsto high-val fruit orchards; and increased useof modern inputs, especially fertilizers. Improved access to health and ei ofacilities inreasedenro4Iment rates in mral education asweUas frequencyofvisits to health care services,and allowedrecruitingof professional personnel to staffschools and health faciites Tebiggest impact was on girls' enrollment i pritmary education,wshich mom than trebldin the project zones duringihestuyp.eriod .... Women also benefite4becausethepavedroads sharplinceased theaffordabli ofbutanefr cokgand heating,raticayreducing womens * dailychore ofcollecting fielwood. Impacts on the envronnt were both positive an negative,.although overall, no environnentay sensitive . areas were putatriskby the road projects,whichdid novolvet new construction. These case studies, focusingon four ofthe ten rural roads improved under the project repr sentwo tmpes (1) pro effects and (2)citical instance. This impactstudy isan exampleofpogrameffectsbecauseitcompared presentcondtions in theareas ofproject roads to the siation priorto ie improvemnents. and tconditions in conpariso roadsdlocatednearby that werenotsubject toimprovementsgduringtheproject period. Thecase stuies are critical instances ofrural roads lending-a better understand ofthe impact of these selected operations wuidbe ofwide .interestbecause ruralr aecount for a significantprportionofBank clientsrural infrastructureprograisand of Banklendingforroads. The sizeofthecaseis aproje-t-heKingdomofMorocc FurthHighway Project (Loan 2254-MOQR)-The sampleroadswereselectedtoachieve geographical distribution inthe north,center,and center-southoftheouantry, tus representingavarietyofcimateagricutural,andeconomic .at --wereobtahied from (1 I surveys conducted atthe fa.rm,regional, and vilage levels; (2) focus groups coniductd $attheselevels,whih provided important quaitative insights forthe analysis ofthe housld-l data; and (3) meetings with representatives from all economic sectors and elected county-level representatives tohelp interpret data and to refine or correefiigsderived from analyses ofthe surveysand focus groups. Source: WborldBank 1998a. The size of the Kenya case study is the city of Nairobi (see box 3). Growing use of case study protocols to organize the data collection has been reported (Yin 1997). These protocols are far Methods of Obtaining and Analyzing broader in scope than a questionnaire, and are particularly Data in Case Studies appropriate when multiple evaluators are collaborating on a number of case studies that are part of the same evaluation. We tend to associate case studies with in-depth interviews. Case studies do involve what methodologists call 'thick descriptions:' tained in a separate case study database-different from the They tend to be rich, full of information that comes from talefinal case study report-has reportedly also taken hold multiple data sources, and often from interviews. But multiple (Yin 1997). methods can be used, from interviewing to first-hand observa- (Yin 1997). tion, to document review. 5 Analvsis of case study data is generally extensive.' This key The study on Mongolia's informal sector merges quantitative and analysis technique used is triangulation. This technique involves qualitative data to understand and quantify the explosion of developing the reliability of the findings through multiple data entrepreneurial activity during Mongolia's transition to a market sources within each type. The validity of the findings, especially economy (see box 4). when trying to determine cause and effect, is derived from agreement among the types of data sources, together with the Selecting the Number of Cases systematic ruling-out of alternative explanations and the n n e explanation of"outlier" results. Examining consistency of In tne idence across different types of data sources is a means of evning verificat er p articular strategies an The right number of instances or cases to select should be based on obkning suchcomprcion. T uher asr patrtni m ba athexplana- the question that is to be answered. There are three general bases for tiong suildinga c thmpa isonsch r s paiew, th chh a n g,e e fu n a selection: convenience, purpose, and probabiity. Using the wrong eaution builingan Themai revlew wehniqussch can graphicn data basis for selecting an instance is a fatal error in case study design. dvsaluayo taXbatSon of einvolvreuehncies, ansdchronologicdalto Only rarelv wiR convenience be a sound basis for decisionmaking, dime series chronolsr. and probability sampling is generally not feasible. Thus, the decision Analyseries ofrcaerstudy dtisgealexnv.'Tskyhis usuadly which variety of purposive site selection is appropriate. 6 Case Study Methods Explanatory Case Studies . . . ...................... .................S Extensive or "thick" analysis Analysis ofmultiple types of data Program Implementation Case Study sources, such as: In evaluation, we frequently need to know whether programs - Interviews with all relevant and projects are being implemented as intended or designed, persons what problems have been encountered, and what adaptations -Observations overtime were made and why. We may want to do such a study when a -Participant observation given project is failing in order to provide a picture of what -Documents happened over time and what might be learned from the - Archives experience that could be applied to other projects. Or we may - Physical information. want to look at a project that appears successful in achieving its desired outcomes, also in order to learn from it and apply the Analysis via triangulation of-data Analysisthrough: lessons to other projects. We may want to represent diversity, the - Pattern matching best and/or worst cases, or typical cases. - Explanation building - Thematic review. -_Thematicreview.* Program implementation case studies typically use a purposive Comparison of evidence for Analysis through techniques such as: sample. The number of cases is dependent on variability across consistency - Matrix of categories the sample, and whether generalization is sought. This type of - Graphic data displays case study relies heavily on published documents and observa- -Tabulation of event frequencies tion. Because they generally require multiple sites, training -Chronological/time series issues and supervision for quality control become issues. ordering. The involuntary resettlement cases provide rich detail on the implementation of the World Bank's policy on involuntary resettlement (see box 5). A few additional words on probability samples are in order. A probability sample is one in which all members of the popula- Program Effects Case Study tion have a known and equal chance of being selected. It is the Case studies are appropriate for determining the effects of programs method of choice for answering questions about how much or or projects and reasons for success or failure. OED does most how extensive a problem is in a population. A problem with its impact evaluation case studies for this purpose. The method is often use in case studies is that the laws of probability operate on large used in combination with others, such as sample surveys, and there numbers-fewer than 30 instances do not always provide the is a mix of qualitative and quantitative data. Analysis usually generalizability to the population as a whole that probability identifies emerging themes. Site selection depends on program samples promise. Some case studies have involved 30 or more diversity; this method should not be used in cases where there is sites selected on a probabilistic basis. great diversity among sites or projects. Its use with the best of projects, the worst of projects, or a typical project is appropriate. Case Study Types and Site Selection The case study on small and medium industries in Sri Lanka This section provides a more in-depth look at the case study studies the effect of Bank lending to small and medium enter- types, the evaluation questions they can answer, design features, prises (see box 6). potential problems, and relation to site selection. 7 Selecting the Number of Cases Convenience Is this site selected because it was expedient for data collection purposes? What is happening and why? Purpose Bracketing What is happening at cxtremes? What explains such differences? Best cases What accounts for an effective instance? Worst cases Whv isn't it working? Cluster How do different types of instances compare with each other? Representative I n instances chosen to represent important variations, what is the situation likc and whv? Tvypical In a typical site, what is happeniing and why? Special ionterest In this particular circumstance, what is happening and why? Probability What is happenings s ra whole and why? 8 Box 5s Recent Experiesce with Involuntary Resettlement This studLyassessestheresettlement process of Bank-supportedprojects and determines the impact on involuntarily displaced people. it consistsqof egt projects from sixcountries: BrazilChinajndia,Indonesia,Thailand,and Tog Two projctswere incuded from India a from Chiiia. Theprincpal findings are mixed. With respe to results- (1.)Compensation rates and schedul-esrangeVidely,butthetrend appears tobetobward: improveements in fairiness andtimeliness. (2)Although uneven, the recodwith reocation is satisfactory. (3)The recardon restoring-letalone . uinproving-incomes has been unsatisfactory. (4) Improvements in physial infrastructure are evident everywhere but,wit thte exception ofChina and Tailand, there are frequentcompaintsabout the operation and maintenance of these structures. (5)At five of te eightsites, loal,sexpre,ssed,,,,' dissatisfactionwiheirrespeiv resetlementprojects Bank performanceis betterthan inthepreceding decade but notas goodas expected; i commitment to the principles of resettlement is visible. Bormwer performance ranges rom better than the Banks policy standards to unsatisfactory Te eigt cases seeced fortissir-counryimpact stud represent fourtypes of casestudies:. - 0:: * Illistrative-des thepolicywithdetailsoftheprojects. *. C.riticalinstance-examineeight instancesoftheapplicationofresettlementpoliy to mitigate the impact of involuntaryreloceation.. Programiiplementation-investigatetheoperationoftheprojects. - Programeffects-examinecausalityortheoutcmeoftheBank's relocationpoliy.: Thesizeisapolicy-involuntaryresetdement, The projecselection was limitedtocasesofinvoluntaryresettlementassociatedwith thecons.truc- tion oflargedamsbecause theyaretheclassic,best-known nase of resettlemerit,aid theBanes resettlement polici's weredesigned largelywith dams tis report uses multipledata sources-it is based upon a review of projt files, other rlevtant documents,andthe results ofa scionomic.. monitorinprogamin five othesixcountries, withrepeatingsurveysof resettlerhouseholdsandcommunitiesplus a folow-upsureyfor Ol)E and sitevisitslyOIevaluators. Up to 90 percentothe housholds surveyedwere the sameas those in previous surveys. Thus, incomesfthe same households-were compared over severalyears.Thestudybenefited from datagathered forEWs 1993 report,EarExperience with nvoluntaryR esetlemen,bu the studywas limited ::by thelackofdatabefore 1991. The studies doan extensiveanalysisofthemutipe typesofdata at their disposaloften usingthe triangulation mehodto ver4 theevid&encefor consistenc £Case studies areoen usedto understand the reasons behnd aset of circumstancesoroutcomes,but these studies stop shortof askin "why qestionstmphasizingproject results and Bankiand borrower performance fndings instead. Source: World Bank 1998c. Descriptive Case Studies The Illustrative Case Study These case studies primarily describe what is happening and study. Its study of structural and sectoral adjustment why, to show what a situation is like. This is especially useful (Jayarajah and Branson 1995) sampled and reviewed 99 loan to help interpret other data that may be available, such as operations in 42 countries, and provides an annex with case survey data. OED has many examples of this type of case studies of 5 countries. 9 Illustrative case study sites are usually selected as typical or w.~~ W.r14 &~ ~ i~,~s~ta a*ai~ representative of important variations. They provide the realism Xtes hSLl and vividness of anecdotal information. The number is kept small to help keep the reader's/user's interest. Data often include visual evidence. Reports may use self-contained, separate Ba~andimv~Uaiidat~mNndlusmks(s1Ajp) narratives or descriptions. Thccentr4~ndingihat$anks~~tpponorSsIswasooquesuonabiy In using the illustrative case study, the challenge is in selecting at 9ponaoUfstibordinateia ps~ffStlLsnk developiwnt ~ the instances. The case or cases should adequately represent the 4frate$gwtti~ W 4fltkai~gad4~irfcctsepjpcorn~ situation. W here considerable diversity exists, it may not be tisntiBamb4 hecrger1994, (liplayed possible to select a "typical" site. S&in * I V The Exploratory Case Study tc'npmy1(i) cmtributcd tothederelupmnwifampreetlhctive The exploratory case study is a shortened case study, undertaken before launching a large-scale investigation. Its function is to develop the evaluation question, measure, design, and analysis 10 tssi&~asesttidyr~presexitstwotypee pr~axneffeasand .u strategy for the larger study. It is most useful where there is iietidttvaiia.c as itg~.dinga~sfoSMis-fr U considerable uncertainty about what the situation is really like. ~~~~~~~~~ ~~~~~~~ OED frequently uses such case studies as part of the scoping . ibt ul i 6fBahkA6~2i4~5i& work. S~ttkiabt~de¶ukdatWeeectsofb.nlypolcyr~Mmg The study on the World Bank's health and nutrition portfolio, Iainst~xSAI~, ~bSn ~a~.codd gsoere,gr.~ cpkir~although a large-scale investigation in itself, also explores hypotheses atory heothswosiuies,ro Ectadota4lrPhdpphi&and methodologies for further investigation of the HNP sector (see Tognhe thesthree.ses eprnset2Qp6etPsfh~E4bs SMIbox 7). Valauez adL Bmegrillustrate tepenilbenefit o ~~~~~~~~~ ~~~~~~~~ explicitly using case studies in scoping projects (Valadez and 1 rce avexntho4dlogy K ~~~~Bamberger 1994, pp. 297-98). They present a study to evaluate ~~~~~~~~~ ~~~~~~~ the impacts of one housing project in Cartagena, Colombia, on .s4~lofivosnwe*.>9 household income. A major design issue was to determine how benef~iars,v~iktheth~ra1fswdaswwnrtgtpuraw4to define a "household?' A descriptive case study investigating ~tt~iit4Iacrvi*~wth.sniipad~iiii4insttuuns2h 9 this concept was prepared on a small number of families. The *u~d~die~.~~o ibthece4iis endingstudy concluded that many community residents had different set ~4brhet4vi&oM.lsndp*ticialt.t~nac4listiwconcepts of the household than did the evaluators. ~~~~~~~~~ ~~~~~~~~Site selection for exploratory case studies needs to include one site that represents each important variation to make a conve- Source: WbddBank 199Th, ~~~~nience sample acceptable. The exploratory case study is often used to investigate cause and effect evaluation questions. It is important in exploratory case studies not to yield to the temptation to make the exploratory 10 Box 7: Lessos fromoExperience in HN;; This study describesandassesses the HNP tfi as p fOED'songoingeflbrt btoean about thedevelope ffectiveness of the BankV a aivties i thest Itinclu fo rasestudies: Brazil,i n Ind ( Theprincipalfindings about Banklendip irethat athoughtBank has designed i thedemandfirhathandthielocalinstitutional potienvironmens i neai i td fi venti . out rai of 1 hcto i the completed ortliooWxetie o othersocia secto r (peranasatisfctor e percentg fNPprojects a re rated assstainable tc pletian -oly 21 pe fNPpr acesit a v The cases used in this studyem two tsexplorato nt at Aso se tk hw t muur~hotproramimplmenatin. A prgrnt~imkmeturatincnestdisAthe ou plaurnhohryplcastatiouyfNkhi?edn policy. Becauseofthecploratorynatu ofthe ti prte co s . Cnsia td achiedbyusngacomonanalytaframework.hfizektheR?4_ sector. nstitneasigniflcproportionoft cot 1sihea hce(Maa e)n heloas at f cvuntrsouahysinheakhcare razilandlndia). ....... Te st es utiple data sources-a reviewofWold ? A sector studies and. ie wt ? government officialsiatioter holders; and aseries of pp c for estudy The casestuiesdoan tsvfnlyothemuiplets ta. zas td et o cf th fninrgs~ are consitenidepnet oie reltitve siaofB lendingn the sape cas.s SourcVWxrld Bank. work the basic study data collection, even though the data may The Post-Conflict Resolution case studies are critical instances seem convincing. Such a procedure is premature and confirms of the Bank's policy on this function (see box 8). the investigator's hypotheses, rather than testing them. Combined Methodology The Critical Instance Case Study The critical instance case study, one of the more frequently used The Cumulative Case Study types of case studies, examines one or more sites for one of two This method is new to the field, but it is one in which OED is a purposes. The first is to examine a situation of unique interest. pioneer. This application of the case study brings together the In such cases, there is little or no interest in generalizability. The findings from case studies done at different times. Previously second purpose, used far less often, is when a highly generalized discussed case study methods are cross-sectional-that is, they or universal assertion is being questioned, and we test it through collect information from several sites at the same time. In examining one instance. contrast, the cumulative case study aggregates information from 1 1 !sexera sthes Woletd Btdfeetand'x erienexwthne Pot-oficts Recn strojcts eeio n itae,tgte ihamasooigtedt ses,the cmltvcaestudv iisI lson o o sio i lan r ftore oeralationsfo e Ban ros e tehene inpo47 sudieseetuanly Irtcompeedn hcoad ed9 rs ulee~ts, ,sythdesis.The reostuyoIrrigtion Keyisue analzd System thercormparaei erea analyed ofth uanktinatrela iotFthrinsternbtisnh 1 ct84) t fficiescy ~ ~ ~ ~ ~ ~ ~ ~ ~ ~~Dffclte winhconuotngpcmutiv&aaeidtdie inlud evalutrion.dTe*urpseaddr tosidpkentaif inputDand prolcyss orofa im ecumulative ca seoitud b yn p eerlopinageas nd ustingestandar compottnents acron~onmic asstabiiainc tatcul bebqufii~~afntitativelycasre RstudoraonoolsumIn anscallcasesuies o rhas nglheaklinmroecst asscitedwih iffrece inoucoe masre.. ThesdfBanhod foMir eam le,hdr tandadhoe s etal oftquestions,iwe would,soon K,:4p9s#@v,tpeand:iae,4q entale rthet speicatidon that iaUs coroin siiyetof data (bthe hveaic nciu xtensriveubodya the Baknowedgtrtie. &urntoi qrual intaieande qandtitunativ)e toleped. Apover a5yearplnentaion astuis hydxff htr fpronaierltdtpo sadatutt 1 2 stde rtht, gauysih }eserpeetteuua ty ¢,M M,,M;; pe Qfcaesdy The, cas swyurtrtie celo s ' tdxt! 'oc th Tevraisidtelsicolecfdatdisfferen on. reviewn o ethefnked timbsnal porfolieof were iankitiated, putogtuherwith mpeans of 1 codigthtri ata sense, the cumulative case study is similar to an evaluation across the 47 studies eventually completed. The coded results synthesis. The study of Irrigation O&M and System Performance were analyzed quantitatively (Finsterbush 1984). in Southeast Asia exemplifies this type of case study. Difficulties with conducting cumulative case studies include In OED applications, our analysis has generally been qualitative, inadequate or uncertain quality of original data or its analysis. USAID did an interesting prospective cumulative case study In OED and the WMorld Bank we could greatly strengthen our use evaluation. The purpose was to identify input and process of the cumulative case study by developing and using standard components of economic assistance that could be quantitatively case study protocols. If all case studies of rural health projects, associated with differences in outcome measures. The method for example, had a standard core set of questions, we would soon entailed the specification that a common set of data (both have an extensive body of knowledge. qualitative and quantitative) be collected over a 5-year period as 12 aBx * 9rr*gation O&M and System Performance in Southeast.Asia This tudy assesses the agro-econenic impacts of investments in gravity-fed irrigation schemes in the paddyds of Southeat Assit determine whethRerand howthe aity of oeation and maintean(Q&M)seices-considered to bethekeyintermediate ctivethat ensures the sutanbiitoaldimpoeents-ifenesthesustiaiXlfmtoehpacts. vi The study finds that thecausolit} awssessedis hg reversed in this cse. Given that theyofepoureconomicsandlwios, hese paddy irrigation schemes facedn uncertain forum irovedl&M performancewil noft resethent Astelackofcompetitivenessp y ingdrWes youngerfamily membersoffthe farmsadeoldemembers whosa behi concentrateonbasicsubsistence s socilcapitwillerode and O& stadad aeSu lielt suffEr;; Th,ea suexe ie fourtypes-pogrameffects,programimpleen cumulatNe,aadexploratory. As aipactstudythe report is a program effetdrit irihi dti e on,tei mentati of th in:their respectveori thus exet ytpe;itbritogehrfindin fgs minividalcnrycastsudies ficusing oirition O&MmakingitactmulatiW stu4 The authors are of' tw mindsabtheir fldit thrrig were elativysimiaratalsi sustste that eIe's,1s le mhavewiderapplication;ho-- bease teftdng l i,n the faeo covetonaiwso te,,a,u,th rsareucrtan but eirghraiblr andr.ecomend veigtmas- hypdtheses in.needofaddtionalempiricalwork. Prom thi tepetetdst dis+'ratory T' szehcasei' a fc -aVi' irri*tio schemes - (intbpaddlndofSotha tsiat Sixschteree s,ele edsires:threeinMynaronehiVietaasandrwo i hilan.. : Theyrewidely dispesed in the targetedegion andwrehnfraietywithourgunran grpresitativeness- -The yi issbase n le sources,inclnndingprv4ousPerformanceAudit Rportst(P ) rojectCompletion IReports(Rs), Sa)f Appai eports ( 4Word?DevelopmttRepr (WDRs),stakeholder tera teriews andisiteinspectio& . auditofafloodcnrt.rl.. an-d draing poect at the threesites in r'gas was included i tefiedokthidezirayoviu difre -cesfrom iand similaritesin organ zationlanedcive iess'mpww 'inga. excesswa'WaW O...w.,,T inteview (lastingup tofivehouseach) eewithgroupsandhousls in alfurcountriesandincludedof Iialst theschenie sitesand ertine'nt ' pu .critOn -autoiies, hadeder,raleners. mn,ome,adothrcharacriticirgtortTe ex utnteaviitedtbelsitstootree cnditined the analand cntrok trutu S6rni;aec ciiynloating,ditrfutna.ndn--tintaining th flo ofw an hesrength and wanse s of: -ThrmerQ&M,especiallasmanaged throughinomal asoitos Thestudy oes ocue? cotol cses, and snanystaeensaboutmpac are . ... .gn int basedon compaisnwith aprialpoetions,not with farmerddoriinlcondiions. Some WrldBank l499Sc Participative Evaluation and the Case Study Reporting or Critiquing the Case Study The process of doing case study evaluations has become more We often publish case studies in the form of precis or summaries participative or collaborative (Yin 1989,1993, 1994, 1997). It is no of longer reports. What should we look for in the decision to longer uncommon for case study evaluators to work closely with publish or not to publish? The standards are similar to those for project officials-for example, to help define the questions to be other kinds of evaluation studies. addressed by the evaluation. Such participation is seen as helping to strengthen ownership of the project, facilitate the transfer of 1. Are the evaluatiotn questionis stated clearly and explicitly? The evaluation skills and capability, and promote democratic governance questions investigated as well as the underlying issues should itself (World Bank 1996a). be explicitly presented. 13 2. Is the case study application clearly described? The case study discusses how the study findings and conclusions converge or should describe the application that was selected (critical diverge from other related work. It also specifies any differ- instance, illustrative, or cumulative) and explain why this ences in the interpretation of the evidence among members application is appropriate for the kind of evaluation ques- of the investigative team or reviewers of the draft report. tions that needed to be answered (descriptive, normative, or 10. Are the strengths and weaknesses of the case study identified? cause-and-effect). Where one or more methods other than A good case study discusses the strengths and limitations of the case study are used, the relationship of the case study to the evidence and takes this into account in formulating the other methods is clear, appropriate, and well-described. conclusions. Generalizations are in line with instance 3. WVas the time span of the study long enough to address the core diversity and the method of selecting the instances. issues fairly? A good case study reports how much time the investigation covered in relation to the history of the instance If we are issuing a precis or other shortened form rather than or program. This is asking, in other words, if the study is the full case study report, the methodology may also be summa- sufficient "to provide a comprehensive understanding of the rized, but it should be given in full in the report. In such cases, event as a whole"- one of the key parts of the definition of reference should be made to the availability of the complete case studies. description. The particular form of the case study write-up will 4. Is the basis for selecting cases clearly specified? A good case study depend on the audience and its needs. presents the rationale for selecting particular cases. 5. Are the data collection methods adequately described? If an Case Studies and Teaching Case Studies: illustrative or exploratory case study is being undertaken, and Critical Differences data collection is unstructured, this should be indicated. When multisite approaches are being used, the actual protocol for data collection should be available. Case studies used for teaching purposes have a different struc- 6.pIfrmorethanl fon investigator collecteonshould dt,aehe qualiture than the evaluative case study we have been describing. We 6. If more than one investigator collected data, are the quality mvfeunl att uls aesuiso td umre control procedures described? This would potentially include may frequently want to publish case studies or study summaries oth basesdforeselecting datacolec?Tors, traini provtidd ancude so that others can learn and we can add to the body of knowledge the like, on a given issue, such as resettlement. If the case studies 7. Are the information sources fully described? A good case generally meet the criteria for good case studies presented here, study presents in detail the sources of evidence. Numbers and we should publish these lessons. Teaching case studies go a step positions of people interviewed and extent and nature of further, however, and are usually a subset of case studies. records reviewed and/or situations observed should be evident. The reader needs to be able to determine from the Teaching case studies generally try to illuminate a decision or a information in the case study report how credible the set of decisions-why they were taken, how they were taken, and conclusions are through the appropriateness and complete- what led up to them. In most cases, the problem is described, ness of information sources. with much detail, and the class is given the exercise of coming 8. Are database formations and data analysis techniques clear? up with and justifying the decision. The students may then be Readers need to know how the data were organized and given an opportunity to compare that decision with those analyzed. Detail should be available on the Steps taken to actually taken in the case. Seldom, however, is there a single right reduce and code the data. Analytic procedures such as or wrong answer. triangulation should be explicitly described. trianglationshoul be exlicity descibed.An example is the John F. Kennedy School of Government, 9. Are arguments for and against various resolutions of the Aample is the oh "TKenned oo Aofovnment, evaluation questions presented? If other studies relevant to Harvard University, case of "The Buenos Aires-Colonia Bridge" the issue are available, their results should be presented and (Case No Ct5-97-1400.0) reconciled with the case study findings. A good case study 14 This case recounts the work of a special Argentina- Who is the target audience? Uruguay bi-national commission convened to examine What are the specific objectives for them? the economics of a proposed bridge that would link the What is the specific decision that needs to be made? two counties. The bridge being considered by commis- sion officials in the spring of 1996 was to extend 41 As Dorothy Robyn has pointed out, teaching cases should be kilometers across the River Plata estuary, from Buenos brief, of general applicability, conflict-provoking, and decision- Aires to the Uruguay City of Colonias. It would be by far forcing (Robyn 1986). Regarding the last two points in particular, the longest in the world. The case recreates the problem she notes: as faced by the commission, which had to consider whether it would make financial sense for a private Coinflict-provoking. Controversy is the essence of a good case concessionaire to build and operate the bridge. Specifi- discussion: it engages students; it forces them to think cally, bridge commission staff had to critique the work through and defend their position; and it demonstrates to and recommendations of a private consulting firm whose them that while there are generally no right answers, there market projections led it to conclude that the US$1 are certain questions that are essential to ask. billion bridge could successfully be financed by tolls of US$60 per car. Decision-forcing. Controversy, even in success stories, generally collects around decision points. Many- Another case example was developed by the Kennedy School for the perhaps most-teachers find that all else being equal, a World Bank, "The Problem of Project Assessment in Social Finance: case works better if it leaves those decisions unre- The Local Initiatives Support Corporation (B)" (Case No. CEI-97- solved-that is, if it presents a choice or decision that 1396-0). confronts a manager or analyst without revealing what the protagonist did and the consequences of that action. Public policy cases fall into two main types: "action-forcing" and This principle applies both to cases that are written "retrospective" (Kennedy and Scott 1985; Robyn 1986). prospectively and those that are retrospective. Action-forcing cases try to place the reader into the shoes of a How to write cases. Producing the teaching case study is a government official (or group of officials) faced with a special skill, and there are professionals who specialize in this. problem requiring action. The bulk of the case is devoted to a There are also those-not necessarily those producing the study, description of the dilemma and the available options for who specialize in the teaching of the case study. Those who are resolution: at the end, the reader, like the protagonist, is poised good at producing them are not necessarily good teachers, and at the moment of decision. The fundamental question here is, vice versa. Kennedy and Scott cite five basic steps for writing "What would you do in X's place and why?" case studies (Kennedy and Scott 1985): Retrospective cases tell the whole story-up to and beyond the decision point, and including some account of the 1. Develop the situationi or problem. The most important element in consequences. In this instance, the basic question is "What researching cases quickly, effectively, and economically is having do you think of what X has done, and why?" a clear notion of what the case isfor. Ideally, this means having a strong sense of the particular teaching points the case is intended Criteria for selecting teaching case studies. Having a good to support and how it fits into the curriculum. In order to ensure and exciting case study is only the first precondition for trying to realism and relevance, ideas for case studies should originate turn it into a teaching case study. Key questions that need to be from real problems: as outcomes of investments in health and answered are: nutrition, growth of free enterprise in Southeast Asia, post- * What problem is being illustrated? conflict resolution, and the like. * What are the underlying issues? 2. Research the case. Case research usually proceeds in two 15 distinct stages after a topic is chosen: library work, followed Will there be a group discussion during a training session, by interviews. The library stage is an opportunity to learn or will learners receive feedback individually? quickly and cheaply. The interview stage is an opportunity Will the assignment be completed during the training for the now informed writer to learn, from the case's principal session or will learners complete it outside of class? actors, the details of the story that are particularly relevant to Will the learners simply report answers, or will a more the case's focus, the actors' own insights into and reflections formal presentation with supporting media be required? on their significance, and the anecdotes that color, spice, and Will each group have the same case, or will different illuminate what might otherwise be dry, dusty history. problems be tackled by different groups? 3. Write the case. Cases vary in length and level of difficulty; Will learners require reference materials to complete the there are no absolute guidelines. It is advisable to adhere to assignment? the following principles when writing a case:3 Will additional research be required, or will learners have * Objectivity: do not analyze and do not editorialize. all the necessary information to answer all the questions? * Disguise. This may be necessary to preserve confidential- ity or to synthesize a single case from several projects, but How to teach cases. In a typical group instruction format, a disguise only when necessary. case is presented to learners who work individually or in small * Worksheets are helpful in the early stages of the course groups to analyze it and answer a series of related questions. The but should not be included later, when students need to questions are usually open-ended and have more than one provide their own structure. correct solution. The work in small groups is followed by class * Exhibits, statistical data, and other relevant information discussion led by the instructor. Austin suggests that the can be included in the body of the text or in appendixes. discussion leader include the following categories of questions * Extraneous material. Some needs to be included in order (Austin 1993). to force participants to distinguish between relevant and irrelevant information. Information seeking: These are the "who, what, when, where" * Leader's guide. It may include material thai is not questions that elicit factual responses that can highlight essential to the case, such as the reason the project was certain particularly relevant pieces of information and selected and a statement of teaching objectives. ensure a common data base. * Pre-testing. Have someone unfamiliar with the material * Analytical: These "why and how" questions provoke review it to make sure that it is understandable to a cold diagnostic, causal, and interpretive thinking, which is often reader. central to achieving mental skill-building objectives. 4. Develop questions, and in some situations activities, to * Challenge: These are "why" questions to force students to support the case. Typical case study questions include extend and deepen their analyses by giving supporting (Sullivan and Ruyle 1997): statements or by responding to counterarguments. * What is the problem? * Action: These "what would you do when, how, and why" * What are the most significant aspects and effects of the questions press the students into making decisions and problem? dealing with implementation processes. * What is the root cause of the problem? Hypothetical: These "what if" questions allow you to create * How could this problem have been avoided? new situations that force the students to extend their * How can it be corrected? thinking under different assumptions; this is a way to go * What can be done to ensure the problem does not recur? beyond certain factual or information gaps in a case. 5. Develop thorough instructions for the case study. These Predictive: These "what will happen" questions force instructions can be developed by answering a series of students to plunge into uncertainty and to substantiate their questions (Sullivan and Ruyle 1997): forecasts. These may be helpful when the instructor knows * Will learners work on the case individually or in small what actually happened and wishes to reveal that after the groups? students have given their prediction, thereby enabling a 16 comparative discussion. Case Study Workshops Generalization: These "what general lessons" questions push the student into a more abstract level of cognitive reason- OED will sponsor workshops on writing and teaching case ing, which is often particularly helpful to achieving knowl- studies and will invite network representatives and other World edge-enhancement objectives dealing with Bank personnel to increase the use of teaching case studies in conceptualization. the dissemination of its work and lessons learned. The instructor needs to develop a plan that includes the topics to be covered and a set of questions. Every detail in the case can be used as evidence in a class debate, so the essence of the instructor's preparation task is contingency planning: analyzing the case for all its possible interpretations (Maister 1981). 17 Endnotes This paper has borrowed liberally from the United States General Accounting Office publication, Case Study Evaluations, Transfer Paper 10. 1.9, November 1990. The authors are indebted to Lois-Ellin Datta, the author of the publication. This book is in the public domain. 1. Yin 1989 contains excellent sections, replete with examples, of case study protocols and means of analyzing case study data. 2. See, for example, Miles and Huberman 1994 for extensive, easy-to-read, within-case and cross-case analysis techniques. 3. Most of the following points are taken from Cohen 1978. 18 Selected Bibliography Agranoff, R., and B.A. Radin 1991. "The Comparative Case Study Approach in Public Administration" Research in Public Administra- tion 1:203-21. Austin, James E. 1993. "Teaching Notes: Communicating the Teacher's Wisdom." Boston: Harvard Business School, Harvard University (5-793-105). Cohen, Bruce. 1978. "Considerations for Writing Case Studies: Course Note." Washington, D.C.: World Bank Economic Development Institute. Feagin, Joseph R. 1991. A Case for the Case Study. Chapel Hill: University of North Carolina Press. Gable, G. G. 1994. "Integrating Case Study and Survey Research Methods: An Example in Information System" European Journal of Information Systems: An Official Journal of the Operational Research Society 3: 2. Greene, Jennifer C., and Valerie J. Caracelli (eds.). 1997. Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrat- ing Diverse Paradigms. New Directions for Program Evaluation, No. 74. San Francisco, CA: Jossey-Bass. Hamel, Jacques. 1993. Case Study Methods. Thousand Oaks, CA: Sage. Harvard University, John F. Kennedy School of Government. n.d. Case No. C15-97-1400.0. Boston, MA. ---. n.d. Case No. CR1-97-1396.0. Boston, MA. Jayarajah, Carl, and William Branson. 1995. Structural and Sectoral Adjustment: World Bank Experience, 1980-92. Washington, D.C.: World Bank, Operations Evaluation Department. Kennedy, David M., and Ester Scott. 1985. "Preparing Cases in Public Policy." Boston: Kennedy School of Government, Harvard University (N15-85-652.0). Klein, Hans. 1995. The Art of Interactive Teaching: with Cases, Simulations, Games and Other Interactive Methods. Needham, MA: World Association for Case Method Research and Application. Lynn, Lawrence E., Jr. 1998. Teaching and Learning with the Case Method: A Guide for Teachers, Instructors, Trainers, and Learners. Chatham: Chatham House. Maister, David H. 1981. "Preparing for Class" Boston: Harvard Business School, Harvard University (9-682-014). Merriam, Sharan B. 1997. Qualitative Research and Case Study Applications in Education. San Francisco, CA: Jossey-Bass. Miles, Matthew B., and Michael A. Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage. Platt, J. 1992." 'Case Study' in American Methodological Thought" Current Sociology 40(1): 17-48. Rice, E. B. 1997. Paddy Irrigation and Water Management in Southeast Asia. Washington, D.C.: World Bank, Operations Evaluation Department. Robyn, Dorothy. 1986. "What Makes a Good Case?" Boston: Kennedy School of Government, Harvard University (N15-86-673.0). Stake, Robert E. 1995. The Art of Case Study Research: Perspectives on Practice. Thousand Oaks, CA: Sage. Sullivan, Rick, and Kim Ruyle. 1997. "Case Studies: An Essential Training Tool" Technical and Skills Training: 27-29. 19 U.S. General Accounting Office. 1990. Case Study Evaluations. Transfer Paper 10.1.9. Washington, D.C. Valadez, Joseph, and Michael Bamberger (eds.). 1994. Monitoring and Evaluating Social Programs in Developing Countries: A Handbook for Policymakers, Managers, and Researchers. Washington, D.C.: World Bank, Economic Development Institute. World Bank. 1996a. Participation Sourcebook. Washington, D.C. 1996h. Kenya: Development of Housing, Water Supply, and Sanitation in Nairobi. Washington, D.C. 1996c. Irrigation O&M and System Performance in Southeast Asia. Operations Evaluation Department. Washington, D.C. 1997a. Korea Impact Evaluation Report. Delayed Development of the Cholla Region: An Institutional Study. Washington, D.C. 1997b. World Bank Support for Small and Medium Industries in Sri Linka: An Impact Evaluation. Washington, D.C. 1997c. Two Water Projects in Malawi. OED Precis. Washington, D.C. 1997d. Transport in Thailand: Lessons for Institutional Development. OED Precis. Washington, D.C. 1998a. Kingdom of Morocco: Socioeconomic Influence of Rural Roads, Fourth Highway Project (Loan 2254-MOR). Washington, D.C. 1998b. The Size, Origins, and Character of Mongolias Informal Sector During the Transition. World Bank Policy Research Workiing Paper 1916. Washington, D.C. 1998c. Recent Experience with Involuntary Resettlement (seven volumes: one overview and six country reports). Washington, D.C. 1998d. Lessons from Experience in HNP (five volumes: an overview and four country reports). Washington, D.C. 1998e. The World Bank' Experience with Post-Conflict Resolution (five volumes). Washington, D.C. Yin, Robert K. 1989. Case Study Research: Design and Methods. Applied Social Research Methods Series, volume 5. Thousand Oaks, CA.: Sage. 1993. Application of Case Study Research. Thousand Oaks, CA: Sage. 1994. Case Study Research: Design and Methods. Applied Social Research Methods Series, volume 5, 2d ed. Thousand Oaks, CA.: Sage. 1997. "Case Study Evaluations: A Decade of Progress?" New Directions for Evaluation 76. San Francisco, CA: Jossey-Bass. 20 ECD Working Papers Barberie, Alain. 1998. Indonesias National Evaluation System. Brushett, Stephen. 1998. Zimbabwe: Issues and Opportunities. Guerrero 0. R. Pablo. 1999. Comparative Insights form Colombia, China, and Indonesia. Mackay, Keith. 1998. Lessons from National Experience. 1998. The Development of Australia's Evaluation System. 1998. Capacity Development: A Diagnostic Guide and Action Framework. ECD Working Papers Anderson, Jock R., and Dana G. Dalrymple. 1998. The World Bank, the Grant Program, and the CGAIR: A Retrospective Review. Other Recommended Reading 1. Mackay, Keith (ed.). Public Sector Performance-The Critical Role of Evaluation. 2. World Bank. 1994. Building Evaluation Capacity. Lessons & Practices no. 4.* 3. ---. 1996. Designing Project Monitoring and Evaluation, Lessons & Practices no. 8.' 4. ---. 1997. Evaluating Development Outcomes: Methodsfor Judging Outcomes and Impacts. Lessons & Practices no. 10.* 5. ---. 1998. Assessing Development Effectiveness. ) i These publications are soon to be available in both French and Spanish. 21