60208 FEBRUARY 2011 Lost at Sea? Navigate! Using Metrics and ABOUT THE AUTHOR Results to Establish the Value of CHRISTOPHER WALSH Communications and Knowledge in Sanitation is the Communications Officer for the Water and Sanitation and Water Service Delivery Program (www.wsp.org), a global partnership adminis- tered by the World Bank, The communications and knowledge work of the World Bank Group takes many where he leads a global team shapes; and although one commonality is its enhancement and support of the of communications specialists from Jakarta (Yosa Yuliarsa), institutional mission and development objectives, communications and knowledge Lima (Yehude Simon), New work is not always subject to monitoring processes as is the case with its operations Delhi (Vandana Mehra), Nairobi (Lucy Wariara Mhina & Toni counterpart. Similar to a ship's captain navigating by starlight, communications and Sittoni), and Washington (Sheryl Silverman, Sara Sultan, knowledge work can be well guided by undefined indicators and tacit knowledge Kara Watkins). He was of what successful communications campaigns and knowledge management look previously the communications specialist with the energy unit like; however, if there were ever a time to develop and use more precise navigational of the World Bank's Africa instruments, now is that time. region, and has also managed communications for Standard & Poor's. This SmartLesson explores how the Water and Sanitation Program (WSP), a global APPROVING MANAGER partnership administered by the World Bank, developed such navigational Jae So, Manager, Water and instruments for communications and knowledge. By explicitly tying communications Sanitation Program, Sustain- able Development Network, and knowledge activities to development objectives and taking time to identify, World Bank aggregate, and monitor outputs, qualitative and quantitative indicators, and assumptions, WSP helped quantify the contribution of communications and knowledge to the achievement of the program's business and development objectives. Background Communications and knowledge are essential components of WSP's work; not only for their In 2008, WSP, a global partnership program support of program activities, but because administered by the World Bank, began to they play a real part in improving services. develop a program-wide results framework, 1 Although WSP has helped bring access to which articulated the links between program improved services to 8 million people in India, outputs and expected results for the 24 Indonesia, Peru, and Tanzania, its real strength countries where WSP works, while also is in articulating the knowledge it acquired offering indicators for difficult-to-measure through that work, and then sharing it areas like capacity building and technical effectively and widely so that the 2.6 billion assistance in helping governments scale up people still without sanitation can also be improved water, sanitation, and hygiene reached. To help demonstrate this connection, services to poor people, thereby helping them the WSP communications and knowledge obtain affordable, safe, and sustainable access team decided to build on this results to water and sanitation services. When the framework and explicitly draw out the global economic crisis started later that year, outputs, indicators, and results that directly putting increased pressure on donors to justify support the program's objectives. and reduce allocations, WSP found its results framework especially valuable in helping Here are some lessons learned from developing donors make the case for investing in and implementing these communications and sanitation and water service delivery. knowledge navigational tools. 1 Co-authored by Sheryl Silverman. SMARTLESSONS -- FEBRUARY 2011 1 Lessons Learned Box 1: Make the Time 1) Find your bearings; identify and adopt the unit's business and development objectives to ensure To ensure synergy with the business unit, WSP's communications relevance of communications and knowledge metrics. and knowledge team identified individual staff to participate in an informal focus group to develop the link between Because communications and knowledge work is often communications and knowledge activities and the program's managed separately from that of technical specialists, objectives. there are often instances where objectives can go The group spent roughly five to seven hours a week for six weeks unidentified, or may be different or too far removed from mapping out WSP's activities, grouping them into emerging those of the business unit. Not only can such incongruity categories, and identifying indicators and responsible parties for lead to difficulties in prioritizing activities and time each. management, it also can call into question the value of The group benefited from meetings with the program's results those activities. specialist, who had led the program manager's initiative to develop a results framework for WSP. The specialist focused the An essential goal of the WSP team was for the discussion by asking questions and evoking thought about communications and knowledge activities to tie explicitly linkage between outputs and clients. and directly to the program's overall business and development objectives in order for value to be clearly articulated. The team did this by identifying both the · Dissemination: knowledge dissemination activities, explicit program objectives, which had been vetted by such as events, Web site, and other multimedia; blast WSP in its recent development of a program-wide results news releases; and media relations. framework, and the level(s) at which communications and knowledge activities contributed to those objectives. · Evaluation and adaptation: surveys and other ways to capture and implement feedback from stakeholders to For example, the principal objective for WSP is to support improve quality. governments in scaling up improved water, sanitation, and hygiene services for poor people. Team discussions · Advocacy: partnerships and activities that contribute to with this objective in mind helped reveal communications sector-related awareness and other campaigns. and knowledge activities and approaches that contributed to attaining it. However, many activities, such as packaging By grouping the activities into four categories, it became and disseminating knowledge, seemed to contribute only easier to draw links to the objective (see Figure 1). indirectly. It became clear that to make a direct connection, which would be essential for relevant metrics, a second- 3) Trust your instruments by listing assumptions and level or sub-objective would have to be identified. To find accepting various levels of a measurement's attainability, it, the team examined the results and intended results of evolution, and longevity to identify both qualitative and everyday communications activities, and concluded that quantitative indicators. these activities helped strengthen the knowledge and advocacy of the water and sanitation sector, which, in To identify communications and knowledge indicators, WSP turn, contributed more directly to governments' scaling examined the activities under each of the four output up of improved services. Finding and articulating a direct categories and asked one question for each: How will we connection opened the door to identifying metrics that know we have succeeded? Through brainstorming and would determine progress in achieving the objectives. iteration, the group identified a thorough selection of indicators. 2) Build navigational instruments by condensing outputs into three or four broad categories. The exercise produced indicators with varying levels of attainability and longevity--that is, it became clear that With activities in 24 countries and globally, WSP has a high capturing 100 percent of the indicators over time might not rate of productivity when looking at the number of be realistic. For example, an indicator such as "the number of advocacy materials, events, and multimedia products it sector interventions identified that use program knowledge" produces each year. Because of the sheer number of combines quantitative and qualitative approaches to try to activities, it was a challenge to specifically identify how capture the impact of knowledge contributions to the sector. they all contribute to strengthening the knowledge and Finding all of these interventions and distilling the portion of advocacy of the sector. the work attributable to WSP would be challenging, to say the least. To meet that challenge, WSP identified and mapped out each activity, and found its connection to the objectives That said, it was important to list such indicators for several (see Box 1). During that exercise, four categories emerged reasons: to provide a more comprehensive list from which to under which each of the activities could fall: select realistic indicators; to give context to the more basic, but consistently measurable, indicators (such as number of · Generation and production: activities such as products disseminated), and to foster awareness of ideal knowledge generation, processing, packaging, and target indicators that, if and when captured, would best branding. measure progress in achieving the communications and 2 SMARTLESSONS -- FEBRUARY 2011 Figure 1. Communications and Knowledge Results Chain knowledge sub-objectives. The team also found that explicitly a way to measure and report on the strength of WSP as a noting the assumptions on which the indicators are based thought leader. greatly contributed to the indicators' credibility and relevance. Below are some examples of indicators at each category and level. Note that for the first year, some indicators had to Further, the exercise revealed activities, such as informal reflect attaining a baseline against which to compare future conversations and networking, that are significant in data. Over time, the team expects that the indicators should achieving objectives at all levels. Anecdotal evidence of evolve and become more refined. impact from knowledge is not often systematically reported and is difficult to capture in a monitoring system. However, 4) Plot the course and dedicate a realistic amount of time WSP identified some qualitative indicators that can be used, to track the indicators. such as an instance where knowledge generated by WSP led to a microfinance solution for sanitation entrepreneurs in Because the list of indicators grew beyond 30, the new Indonesia; and another instance in India, where nationwide, challenge was how logistically to appropriately track the government is rolling out a WSP-produced manual to them. WSP developed a tracking system that plotted each guide villages on ways of improving water quality. indicator in an excel spreadsheet. Roughly 15 percent of one team member's time is dedicated to collecting the These activities are clearly valuable for strengthening water indicators through research and staff correspondence and and sanitation sector knowledge, and are worth tracking as assigning the information to the appropriate indicator(s) in relatively real time. Figure 2. Indicators at the sub-objective or intermediate outcome By assigning a percentage of one staff member's time to level ensure that the indicators are tracked as comprehensively as possible, the program can monitor progress regularly, provide feedback on implementation, and guide future communications and knowledge activities. To maximize the effectiveness of the tracking system, all program staff should become familiar with and buy into the indicators. This is because many of the indicators have to do with the technical work staff perform each day, and therefore they will likely be more aware of instances of results that correspond to tracked indicators. Conclusion In the past, perhaps for several good reasons, communications and knowledge activities have not SMARTLESSONS -- FEBRUARY 2011 3 Figure 3. Indicators at the output level generally been measured against pre- knowledge team found that the framework identified indicators. After all, to explicitly could be used to more effectively and identify indicators that can be used to track comprehensively articulate its work. Team progress in communications and knowledge is members also envision eventually using the to expose an area traditionally perceived as an framework to help strategically prioritize and art to measurements that are more scientific manage its activities. in nature. This can be both frightening and exciting--frightening because naming In conclusion, for WSP, knowledge is the map indicators also opens an activity to failure, and to a massive scaling up of improved water and exciting because it provides an opportunity to sanitation services for billions of poor people. substantively demonstrate progress, success, By finding ways to measure communications and/or lessons. and knowledge work and its impact at various levels, WSP has identified the navigational By taking the time to identify and track tools necessary to help plot the course in the appropriate indicators and corresponding hopes of reaching sooner that elusive assumptions, the communications and destination. DISCLAIMER IFC SmartLessons is an awards program to share lessons learned in development-oriented advisory services and investment operations. The findings, interpretations, and conclusions expressed in this paper are those of the author(s) and do not necessarily reflect the views of IFC or its partner organizations, the Executive Directors of The World Bank or the governments they represent. IFC does not assume any responsibility for the completeness or accuracy of the information contained in this document. Please see the terms and conditions at www.ifc.org/ smartlessons or contact the program at smartlessons@ifc.org. 4 SMARTLESSONS -- FEBRUARY 2011