Water and Sanitation Sanitation in Wonosobo: Two Evaluation Approaches Compared I~~~~~~~~~~~S - _13 L~~~~~~~~~~~1 Water and Sanitation Program tf -> t1'1_..................... - for East Asia and the Pacific Field Note Prepared by: Ruth Walujon S Richard M. Hopkins Arie Istandar - April 2002 s 'I -> > This study was financed by the Government of Australia through /A AID the Australian Consultants Trust Fund executed by the World Bank Photographs by Richard M. Hopkins (WSP-EAP) This document is published by the Water and Sanitation Program - East Asia and the Pacific. Copies are available from the Water and Sanitation Program; Jakarta Stock Exchange Building, Tower 2, 13th floor; Ji. Jenderal Sudirman Kav. 52-53, SCBD, Jakarta 12190, Indonesia. Phone: (62-21) 5299-3003. Fax: (62-21) 5299-3004. E-mail: wspeap@worldbank.org. Website: http://www.wsp.org Sanitation in Wonosobo: Two Evaluation Approaches Compared A recent comparative case study on two program such as latrine ownership and latrine benefits as perceived evaluation approaches found out that both the by the village residents and leaders, the results varied, due conventional survey method used by UNICEF and to differences in the methodology design and sampling the participatory method employed by WASPOLA' yielded process. comparable outcomes in the assessment of UNICEF's sanitation program in sample villages in Wonosobo district, The design of each evaluation methodology was quite Indonesia. This field note presents a summary of the distinct. The conventional surveycollected data by measuring comparison of results produced by the two evaluation individual responses to a series of questions about sanitation approaches. and latrine uses; the survey provided limited discussion opportunities with its respondents to clarify unclear issues Contrary to prevailing belief that participatory methods are or solve problems. In contrast, the participatory method more complex, more time-consuming and more expensive emphasized focus group discussions to help empower the than conventional survey methods, the study showed that participants in identifying, discussing, clarifying, and the total costs involved in the preparation and resolving problems within their own communities. Data implementation of both assessment activities were similar. recorded represented group conclusions reached collectively However, the completion of the participatory method by the participants. required a smaller team of better trained field staff and less implementation time than the conventional survey. Both approaches methods produced diverse data sets because the sampling pool differed. The conventional survey Further comparison of results generated other similarities targeted individual members of certain groups in the village and differences. In the assessment of latrine use, for example, to extract certain information, but it excluded social the outcomes were rather similar although the respective background considerations of its respondents. In contrast, methodology implementation differed. In other assessments, the participatory method aimed to gather information 1 WASPOLA is the Water Supply and Sonitation Formulation and Action Planning Project implemented by the World Bank's Water and Sanitation Program- East Asia and the Pacific (WSP-EAP) in conjunction with the Government of Indonesia, led by the National Development Planning Agency (BAPPENAS), with majority funding by the Government of Australia through AusAID. from focus group discussions where the sampling design took into account the representation of both gender and . socio-economic factors. Data obtained from the two methods therefore reflected the different viewpoints of the sample - groups. Project Background The study was conducted within the context of UNICEF's stimulant latrine package distribution program in four villages in the Wonosobo district of Central Java: Sawangan, Rejosari, Mergosari, and Kagungan. The UNICEF program was aimed at improving the health and hygiene of rural ; ' .;, - residents by encouraging latrine construction at the - household level through the distribution of stimulant packages, typically consisting ofa water-seal closet, a length respective implementation details (see Figure 1). When of PVC pipe, and a sack of cement. Villages were selected program assessment results were different, both the based on the existence of previously implemented approach design and implementation process were government sanitation and diarrhea reduction programs, evaluated to identify the possible reasons for the different the community's potential to support the stimulant program, outcomes. When the assessment results were similar, both and the establishment of at least two community programs, approaches were determined to be providing comparable such as community health centers and children education data of village conditions. projects. UNICEF and WASPOLA agreed to collaborate in conducting Objectives of the Comparative Case Study a case study in the Wonosobo district of Central Java to compare the results of using the conventional survey by * To compare the design and implementation applying questionnaire surveys and the participatory approaches of thetwo evaluation methodologies in evaluation method by using tools derived from the collecting data and producing results. Methodology for Participatory Assessment (MPA) and the * To compare the results of the two methodologies Participatory Hygiene and Sanitation Transformation whenever adequate and suitable results or data are (PHAST). available for comparison. * To focus on the quantitative data comparison since The conventional survey consisted of two phases. The first almost all outputs of the conventional survey method phase, in August-September 1999, included a baseline are quantitative. survey prepared by CIDA2 consultants and implemented by a local NGO to assess village conditions prior to the As indicated in the box, the comparative study of the two distribution of latrine packages. The second phase, in July methodologies analyzed not only the results, but also the 2000, entailed an evaluation survey prepared and design aspects of the data collection processes, and the implemented by the same local NGO to evaluate village 2 CIDA = Canadian International Development Agency 2 conditions and program results after the package Research Process disbursement. Both the conventional survey and the participatory methods The participatory evaluation method was implemented once employed field personnel for methodology implementation. in April 2001 to assess program results and conditions. Tools In the conventional survey, enumerators equipped with for the participatory evaluation activities were designed by questionnaires interviewed respondents at the household WSP-EAP staff and implemented by a local NGO. level. In the participatory method, field staff facilitated Comparative Analysis Flowchart Step IComparison of Result Comparison of Results * If similar, then list results. * If different, then proceed to Step 2. Similar Different Results . Results Step 2 Comparison of Comparison of Methodology Design M D * If the methodology design used was invalid, then such application most likely caused the results disparity highlighted in Step 1. * If methodology design used was valid, Valid Invalid but results remained totally different, Methodology Methodology then proceed to Step 3. 4 COmpari son of Methodology Implementation Step 3 Comparison of Methodology Implementation * If the methodology was improperly applied, then such conduct most likely caused the Proper Improper results discrepancy noted in Step 1. Implementation Implementation * If the methodology was properly used, but results remained totally different, then the change in village conditions | Verification of most probably caused the discrepancy. Village Condition 3 interactive and participatory assessments,by community respondents, thus allowing little flexibility in clarifying any groups, and organized focus group discussions. vague questions or uncertain responses. The participatory method was designed to gather How Was Information Collected? information from the village residents' groups segregated by gender and socio-economic levels. The exact number of Enumerators of the conventional survey interviewed participants was not as crucial as the representativeness of respondents including village leaders, heads of the groups. In some exercises, women's participation was household, and school children using a standard more significant because women represented the questionnaire. The survey was also designed to cover beneficiaries; in other activities, participation of the various an equal number of male and female respondents. The poor-rich and men-women groups in the village was more responses reflected individual preferences. important. The participatory method provided ample time for discussions with participants, and offered opportunity to In the participatory method, information was collected include various responses which were locally relevant. The from focus group activities and discussions as well as method relied on group participation and the recording of individual interaction. The method allowed every group conclusions, thus minimizing individual respondent community memberto express his/herviews and discuss biases. issues, and opened opportunities for collective problem- solving and decision-making. The method also included observation at and interaction with individual households to verify the collected information, such as the actual use and maintenance of household latrines. By design, the conventional survey targeted as many L v - households as possible and balanced the respondents by gender, but not by socio-economic groupings. Households I were selected randomly from a list of village residents in each respective village provided by the Village Chiefs; it was assumed that each household was readily represented by one household member. Three different questionnaires were developed to cater three different respondent * categories: village leaders/cadres, community/ordinary I !-- - , residents, school children/students. The data collected were highly sensitive to who participated in answering which questions in the questionnaires. By nature, the survey - implementation provided minimal discussion with the -- - 4 The participatory method at times encountered difficulties Different Designs, Similar Results in implementation. For example in some of the exercises, - - the participants were not fully representative of the village Both methods evaluated the use of sanitation facilities because only certain groups, like the rich, attended the installed after the latrine packages distribution. Although discussions. Data generated from these exercises were each method used a distinct approach to assess latrine use recorded and identified during the course of the study. among the village residents, the results were comparable. Emphasis on Equity In the conventional questionnaire survey, latrine use (referred to here as use of toilets or neighbor's facility) questions were The pariic,poior meilhod iends .isel! lo promoting equity linked to ownership. During the survey implementation, the in community voice through iis focus group discussions enumerators assumed that well-maintained latrines observed and communiry ascessmnent aci-,les, with separate in the households were synonymous with effective use; there men's and women's -groups, arnd poor and non-poor was no discussion to clarify whether the clean latrines were household groups.- By.doing- so, the.design and used by all or only selected household members. implementation of sanitation -p-rogramns- can be more Respondents were limited to choose their answers from a focused, not only on prorviding' ihe sanitation facilities focused, not only on providing h- sanioton facilitis .preconceived list of latrine use responses prepared by others. and services that the people .,anl' but alsb on promoting and services that the people n bu aloon prSurvey results in three of the four villages indicated that use equitable access to improved- sonif6fion within the equitable access to improve- sanitation wti th ':.'of private, public, or a neighbor's toilet was overshadowed community. - communit. -. -by the use of rivers and fishponds (see Figure 2). Use of Sanitation Facilities after Package Disbursement: Conventional Survey Toilet Use of neighbor's facility FE] Sawangan Public toilet l *l Rejosari ; - i g Kagungan River / pond F . Open space / bush X iF 0 20 40 60 80 100 % of Total Respondents 5 In some respects, the results of the participatory evaluation | were similar although the participatory method yielded more information on the extent (or lack thereof) in community sanitation behavior. The participatory exercise asked men I and women participants which facilities they used before A and after the package distribution. In the exercise, the - / / ,,f facilitator's interactive role was crucial to obtain honest and representative responses. Site visits were then performed to / , confirm the data. Figure 3 summarizes various types of - - sanitation facilities used by the participants. In the participatory method, the greater variety of answers resulted from the participants being able to identify and discuss freely the facilities they used; the participants were together significant sites such as rivers and fishponds (which not restricted to a pre-determined set of answers and instead have different environmental and behavioral implications came up with eight different sites. The questionnaire survey for sanitation programs). While this restriction highlights the restricted respondents to only five possible sites in the same inherent difficulty in designing questionnaires, at the same communities and lost important information by lumping time it illustrates a strong advantage of the participatory -__lUl:ff | Use of Sanitation Facilities Before and After Package Disbursement: Participatory Method Before After Toilet inside I I the house Toilet outside _ the house Public toilet Pit latrine Fish pond l Open space/ bush l 1 - -- - 23 River _ - _ _ _ __ 100 80 60 40 20 0 20 40 60 80 100 % of Total Participants nl Sowongan El Mergosari [3 Rejosori El Kogungan 6 method generally in allowing participants the freedom to conventional survey only obtained information on the current describe their outcomes, and thus the structure of the behavior pattern. For instance, although toilet use increased evaluation results. after package distribution and fishpond use decreased, fishpond uses remained common. The survey results, however, were similar to the 'after package distribution' results of the participatory method. Both methods revealed that fishponds were commonly used, both directly Different Designs, Different Results and as the final discharge points of latrines instead of septic tanks or pits. Fishponds were generally used more frequently Both approaches assessed latrine ownership after package than privote (inside and outside) and public toilets in three distribution by interviewing village residents in the of the four villages (after package distribution). conventional survey, and engaging them in focus group discussions in the participatory method. The assessment In addition, the participatory method produced a comparison results from the two methods were different, but they were of community sanitation behovior before and after latrine consistent in identifying villages with the lowest to the package distribution, whereas (at least in this case) the highest latrine ownership (see Table 1). Table 1 Comparison of Latrine Ownership Ownership (%) Village Conventional Survey Participatory Method After Rich Middle Poor After (Total) Kagungan 12 21 6 0.5 7 Rejosari 38 53 31 9 16 Sawangan 48 93 34 3 30 Mergosari 55 100 77 46 59 The discrepancies in latrine ownership results were attributed usage; those who answered "No" were not asked further to the different methodology design. The conventional questions. The survey questionnaire did not leave much room evaluation survey was insensitive to the socio-economic for discussion to clarify latrines that were "owned but not backgrounds of the village households. Consequently, the installed" vs. "owned and installed". While the survey's intent survey may have unintentionally targeted more non-poor was to collect data of owned latrines, meaning installed respondents who typically already owned latrines, thus and used, it did not consider instances of residents who resulting in a higher ownership rate in three of the four received latrine packages but had not installed the latrines, villages. and thus had not used them, or those who installed them but instead used the public or their neighbor's latrines. As a In addition, the survey used questions that were closed-ended result, respondents who owned latrines, whether built or and multi-staged. Survey respondents who replied "Yes" to not, responded positively to latrine ownership, thus latrine ownership were further questioned about latrine increasing the ownership rate. 7 The participatory method discussed latrine ownership and in most villages, they had the lowest access to latrines. Due use issues among village residents in focus group discussions, to its design, the conventional survey was not able to capture and assessed household demand for latrines through cost/ this information. benefit perception assessments. Vague ownership definitions were discussed and clarified to avoid any misinterpretation. The method also considered local conditions as part of its Different Designs, Different Results design, in particular the different socio-economic groups within a village, and distinguished recipients of the UNICEF Both approaches gathered data on the user's perceptions program from other development projects and from those of the benefits of owning and using latrines. The results who built the latrines all by themselves. Ownership data arising from the two methods however were different; were then verified with actual household visits. Results from community perceptions obtained through the participatory the method suggested that different socio-economic groups method were more varied. In the conventional survey had different rights and privileges, and received different method, only village leaders/cadres were interviewed about benefits, although they were part of the same community. benefits of latrines after latrine package disbursement (see Although the poor formed the major socio-economic group Figure 4). 1 ;; -Benefits of Latrines as Perceived by Village Leaders: Conventional Survey Hygiene improvement Clean environment _ ___ zi-I Proper defecation site Increase household- owned latrines Increose of diarrhea knowledge _ Do not know 0 25 50 75 100 % of Total Respondents |E Sawangon [l Mergosari l Rejosari E3 Kagungan 8 According to the respective village leaders/cadres of the The participatory method involved a cross-section of thevillage survey, greater knowledge of diarrhea and cleaner residents, namely the poor and women, to express what they environment were the two most common perceived benefits. perceive as the benefits of owning and using private latrines. Field assessments, however, documented that most village The participants not only had to identify the benefits, but they residents continued to use fishponds as the final discharge also had to rank them (see Figure 5). points of private latrines instead of septic tanks or pits after the latrine packages were distributed. Using fishpond water Participants ranked the following benefits from most to for bathing/washing is a common source for the transfer of least: better hygiene, better for health, cleaner environment, diseases. Contrary to what the village leaders noted, easier access (proximity), greater privacy, greater comfort residents seemed no more aware of the causes of diarrhea and safety (protected from rain, ample space), and after package distribution than before distribution. convenience/better image for guests. -tq6B-s | Benefits of Latrines as Perceived by Village Residents: Participatory Method Hygiene improvement Clean environment l l l I _ _~~ ~ ~ - ., -. .s _____ i- I Proper defecation site l _i Sawangon Better for heolth e-_ l i O Mergosari - I I I __ I [ | O Rejosari Comfort and safety -r 7_ - i . C~~~~~~~~~ Kagungan Easy access a Privacy I I Convenience/better imoge for guest _ o 1 2 3 4 5 6 7 8 9 10 Ranking (1-10) Note: Ranking Process: once the perceived benefits were mentioned in separate men's and women's groups, they were ranked with a scoring system between 10 (highest) and 1 (lowest) using beans or seeds. The ranking scores in Figure 5 represented the average scores of both gender groups. For instance, men and women in Mergosori ranked privacy as the highest benefit of using latrines relative to those in Kagungan, who did not rank privacy as a main benefit. 9 The disparity of the results can be attributed to the sampling respondents had to choose their responses from regarding process and the methodology design, namely the research the benefits; they were restricted to the provided answers. tool selection. To identify the benefits of latrines, the conventional survey asked only village leaders/cadres while Relative to conventional survey results, the participatory the participatory method asked a greater cross-section of method outcomes were more diverse due to the more varied residents, particularly the poor and women. Assessment nature of the participants. The participants had full freedom resultsindicatedthatvillageleadershaddifferentperceptions to mention the kinds of benefits they perceived, which relative to the residents. Also, the survey questionnaire generated a more varied range of responses; they were not provided a pre-determined list of answers from which the limited to choose from a pre-determined list. Table 2 Comparison of Costs and Resources to Implement the Conventional Survey and Participatory Activities Parameters Conventional Survey Participatory Method 1 national consultant, 48 person weeks 48 weeks 1 survey coordinator, 12 person weeks 1 coordinator, 5 weeks 5 person weeks 12 weeks STUDY 2 field coordinators, 40 person weeks TEAM 20 weeks & 16 field researchers, 48 person weeks 7 field researchers, 14 person weeks STUDY 3 weeks in the field 2 weeks in the field PERIOD 4 weeks for material 76 person weeks 1 week for material 8 person weeks and field preparation, preparation and training and training (19 persons) (8 persons/team) 5 weeks reporting 15 person weeks 2 weeks reporting 14 person weeks (by 3 coordinators) (7 persons) TOTAL INPUTS 239 person-weeks 41 person-weeks TOTAL COST* 100 million 82 million (Rp.) Note: * Total cost was inclusive of reimbursables (travel), material preparation, training, field implementation, data consolidation/analysis, and report preparation. In addition, the costs and resources of the external consultant who prepared the baseline survey questionnaire were assumed equivalent with those incurred by various WSP-EAP personnel involved in the initial planning stages of the comparative study. These costs were omitted from the total costs of both methodologies. 10 Despite the differences in overall results, both the village The conventional survey consisted of baseline and evaluation leaders and residents interestingly perceived better health surveys that were implemented at two different time periods; and a cleaner environment as important benefits of owning the baseline survey was an inherent part of the entire survey. and using latrines. The survey incorporated fees for a local consultant to perform project preparation and design tasks to aid UNICEF in the preparation of detailed project Terms of Reference for the Different Methodologies, participating NGO (for baseline survey), design of detailed Comparable Costs baseline survey questionnaires, and upfront contract preparation. In the evaluation survey, data were estimated from the baseline survey parameters. Based on the smaller The total costs to prepare and implement the conventional number of participants in the evaluation survey relative to the baseline, it was assumed that the overall evaluation Table 2 provides a summary of the overall inputs and costs survey preparation and implementation required 2/3 of the of implementing the two methods in the four villages in time needed for the baseline. the Wonosobo district of Central Java. It shows that the conventional survey used more, but cheaper enumerators, The participatory method was conducted once and its total whereas the facilitators for the participatory method were cost did not significantly differ from that of the conventional more expensive, but smaller in number and better trained. survey. *\' ..e1.,X , 4- .% ~ ~ Key Findings * The case study shows the two approaches produced both similar and different assessment results. However, it is also clear that the use of different approaches can provide comparable results if sufficient attention is given to their design and implementation processes, particularly with respect to the sample selection and the questionnaire development. a The sampling process was a key attribute in the discrepancies of the assessment results. The participatory method was sensitive to the representation of men-women and rich- poor groups in the respective communities to reflect differences in village conditions, such as latrine ownership, use, and benefits. The conventional survey results were sensitive to the total number of respondents in order to properly represent the village conditions. Identifying a representative sample requires a lot of effort, especially when village households show more diversity. Therefore, by excluding considerations of village- level variations during the design of the sampling process, assessment results may be biased toward certain groups within the village. * The participatory method encouraged village residents to voice their ideas, opinions, issues, and add local knowledge to the response categories, thus resulting in a broader picture of sanitation conditions in their villages, such as latrine benefits and latrine uses. The conventional survey entailed a pre-determined set of response categories, thus restricting the respondents' preferences to those listed. In theory, this problem in the conventional survey technique could be overcome by extensive field testing of the survey questionnaire. But in practice, as the study shows, inadequate time was invested in undertaking this part of the survey, resulting in a less accurate picture of village conditions. * The total costs of the participatory method were comparable to the costs of the conventional survey. The conventional survey employed more, less expensive enumerators, whereas the participatory method used fewer, better trained facilitators. The participatory method required less time to implement and was easier to manage because of its one-time application to assess both before and after village conditions, whereas the conventional approach required two separate surveys (baseline and evaluation) to measure and assess the pre- and post-project conditions. 12 Water and Sanitation Program East Asia and the Pacific Jakarta Stock Exchange Building, Tower 2, 13th floor JI. Jend. Sudirman Kav. 52-53, SCBD, Jakarta 12190, Indonesia Phone: (62-21) 5299-3003, Fax: (62-21) 5299-3004 Website: http://www.wsp.org WASPOLA Project Office E-mail: waspola 1 @cbn.net.id, Website: www.waspola.org /AusAID This activity and publication were funded WASPOLA, a policy initiative with suppo from AusAID The Water and Sanitation Program is an international partnership to help the poor gain sustained access to improved water supply and sanitation services. The program's funding partners are the Governments of Australia, Belgium, Canada, Denmark,- Germany, Italy, Japan, Luxemburg, the Netherlands, Norway, Sweden, Switzerland, and the United Kingdom; the United Nations Development Programme and the World Bank