69097 Assessment of the delivery of the Vietnam Development Report 2010 – Modern Institutions and its impact on policy debates around institutional reforms in Vietnam Final Report Arnaldo Pellini, Vu Phuong Thao, and Nguyen Le Hoang July 2011 1 Table of Contents Table of Contents .............................................................................................................................. 1 Table of Abbreviations ....................................................................................................................... 3 Executive Summary ........................................................................................................................... 5 1. Introduction .................................................................................................................................. 7 2. Analytical framework ................................................................................................................... 7 3. Research activities ...................................................................................................................... 8 4. Analysis of uptake of the VDR 2010 .......................................................................................... 10 4.1 Downloads and Internet traffic on World Bank websites ........................................................ 11 4.2 Citation analysis ..................................................................................................................... 14 5. Analysis of outcomes of the VDR 2010 ..................................................................................... 16 5.1 The preparation of the VDR 2010 .......................................................................................... 18 5.2 The quality of the VDR 2010 .................................................................................................. 18 5.3 The communication strategy of the VDR 2010 ...................................................................... 19 5.4 Contributions of the VDR 2010 .............................................................................................. 20 6. Conclusions and recommendations .......................................................................................... 21 References ...................................................................................................................................... 25 Annex 1 – Uptake data .................................................................................................................... 26 Annex 2 – Semi-structured interview ............................................................................................... 29 2 3 Table of Abbreviations CECODES Centre for Development Studies & Community Support GAPAP Governance and Poverty Policy Analysis and Advice Program TCER Training Centre for Elected Representatives VDIC Vietnam Development Information Centre VDR Vietnam Development Report VDR 2010 Vietnam Development Report 2010 – Modern Institutions 4 5 Executive Summary As Vietnam’s development history has shown, the complexity of development processes increases over time. Some respondents to this study have highlighted that since 1975 this increased complexity has become increasingly more evident in Vietnam. The post-war period focused chiefly on reconstruction and food security. This was followed, in the mid-1980s, by socialist–oriented market reforms, which are still ongoing today, but have increased in complexity. In order to continue to sustain these reforms Vietnam is looking to establish modern institutions to manage the process. It has been said that the next ten years will be the decade of governance reforms and modern institutions in Vietnam. The increasing complexity of the development process goes hand in hand with an increasing demand for knowledge and analytical products, which provide the evidence to inform policy changes and reforms. The Vietnam Development Report 2010 – Modern Institutions (VDR 2010), which was published in December 2009 by the World Bank and 14 other donors, represents a joint statement from the donor community on progress and prospects for Vietnam’s development of modern and accountable institutions. Like other previous editions of the Vietnam Development Report series, which started in 2002, the VDR 2010 aims to influence the reform agenda, build consensus among thinkers and the public, and harmonise the donor position on key development issues for Vietnam. The influencing effect of a report such as the VDR 2010 is linked to its quality. However, while quality is necessary it is not sufficient to secure influence on policy debate. The preparation and communication strategy of the report are of equal importance in both the short and long term. The VDR 2010 was first presented at the Consultative Group meeting held in Ha Noi in December 2009. The overall opinion of the VDR 2010 gathered for this study is very positive, including the quality of its analysis, the historical perspective on governance, in addition to the description of the specific challenges that lie ahead for the country. A thorough review approach, which involved internal reviews within the World Bank, from participating donors, Vietnamese experts who acted in their personal capacity, and two international experts ensured the high quality of the report. The consultation process that led to the production of the VDR 2010 was greatly appreciated by participating donors, because it allowed them to directly contribute with knowledge products, as well as the open process of giving comments and receiving feedback. Moreover, the report not only allowed donors to present a joint message, but the consultation process also provided a useful space for sharing and exchange between the organisations involved in the process. The communication and dissemination approach of the VDR 2010 involved the distribution of 4,000 copies of the report to national and provincial institutions, two thirds of which were produced in the Vietnamese language. The new implementation of dissemination workshops, conducted at national and local level and organised in collaboration with national institutions and a local NGO, were among the most successful outputs of the strategy and allowed government counterparts at national and sub-national level to prepare comments and make public their views on the content of the VDR 2010. The analysis of uptake of the VDR 2010 was more difficult than expected, but it found that the report has been quoted or used as a reference in donor programme and policy documents, by researchers and scholars, and by some government institutions. The overall aim of the VDR 2010 is to provide information and ideas that can contribute to a broader discussion around governance and institutional reforms in Vietnam. The process adopted for the VDR 2010, which combines various rounds of consultations between the participating donors, a thorough review approach, and a communication and dissemination approach at national and local level, has resulted in the VDR 2010 being used as reference and background publication for a number of donor interventions and as a reference in policy discussions. The VDR 2010 is therefore achieving its influencing objective. While the monitoring of the uptake of the VDR 2010 could be improved with simple and cost effective online solutions, the quality of the analysis of the VDR 2010 will remain relevant for the coming years. 6 7 1. Introduction The Vietnam Development Reports (VDR) are a series of reports, which have been published by the World Bank Resident Mission since 2002, and focus on key development issues for Vietnam. The VDRs aim at tapping into existing knowledge, achieving an ‘upstream harmonisation’ so that donors can engage with the government through a common message, and utilises reviewing committees consisting of Vietnamese experts who are employed in their personal capacity to help to improve the understanding of the policy environment in Vietnam. The timing of the VDRs has always been planned to have a six months interval between the Poverty Reduction Support Credit process and the release of the VDR at the end of the calendar year. The Vietnam Development Report 2010 – Modern Institutions (VDR 2010) – examines governance and the need for the development of modern institutions that can sustain Vietnam’s recent entry into the group of Middle Income Countries. In particular, the VDR 2010 analyses the interaction between citizens, the private sector, and the state at a national, as well as local level, and from a historical perspective. Furthermore, the VDR 2010 explores devolution and the characteristics and nature of the required upward and downward accountability. The VDR 2010 was launched at the yearly Consultative Group meeting held in Ha Noi on 3rd and 4th December 2009 and is co-signed by fourteen donors. It is also a key deliverable of the DFID trust-fund ‘Governance and Poverty Policy Analysis and Advice Program‘(GAPAP), which is managed by the World Bank. GAPAP has focused on developing three specific goals: (i) Vietnamese institutions capable of collecting, analysing and monitoring poverty and governance data; (ii) Systems and processes for decision-making, including access to robust analysis; (iii) Strategic and policy direction of World Bank informed by GAPAP: identified as a priority, as the Bank moves towards non-concessional lending. Since the VDR 2010 is just sixteen months old, the present study centres on the delivery of this analytical product and its impact on policy debates around institutional reforms in Vietnam. As recognised in the terms of reference for this study, institutional reforms are a long-term agenda and the VDR 2010 is one product of GAPAP. The emphasis of this study is therefore to assess the influence of the report on discussion and debate in Vietnam. This study is organised as follows: in the next section we describe the analytical framework that has informed the study approach. Section three describes the research activities. Section four provides an analysis of the uptake of the VDR 2010. Section five includes the analysis of outcomes of the VDR 2010. Section six summarises our main conclusions and provides some recommendations. 2. Analytical framework Policy influence is the primary aim of many donor interventions in Vietnam including a study such as the VDR 2010. Firstly, it is important to clarify what type of impact or influence is sought. Broadly speaking, policy itself can be defined as ‘a series of documents and decisions that are best described as a set of processes, activities or actions (Neilson 2001 in Jones 2011). Jones and Villar (2008) draw on Keck and Sikkink’s (1998) to highlight five key dimensions of possible policy impact: • Framing debates and getting issues on the political agenda: this is about attitudinal change, drawing attention to new issues, affecting the awareness, attitudes or perceptions of key stakeholders. 8 • Encouraging discursive commitments from states and other policy actors: affecting language and rhetoric is important, for example promoting recognition of specific groups or endorsements of international declarations. • Securing procedural change at domestic or international level: changes in the process through which policy decisions are made for example opening new spaces for policy dialogue. • Affecting policy content: while legislative change is not the sum total of ‘policy change’, it is an important element. • Influencing behaviour change in key actors: policy change requires changes in behaviour and implementation at various levels in order to be meaningful and sustainable. A number of activities can be undertaken in order to influence policy. Harry Jones (2011) draws on Start and Hovland (2004) to distinguish between an inside track approach which involves close collaboration with decision-makers and an outside track which seeks to influence change through pressure and confrontation. In addition, on the vertical axis, evidence and research or values and interests can lead the influencing activities. Figure 1 below illustrates four approaches to policy influencing: Figure 1 - Policy influencing approaches Based on these definitions and the policy influencing approaches described above, it can be argued that the policy change aim of the VDR 2010 is to influence by framing debate and getting issues on the political agenda. It does so through a mix of advisory and advocacy approaches, which includes the generation of evidence and cooperation not only with the Government of Vietnam, but also with the donor community at large. 3. Research activities The present study is concerned with the influence that the VDR 2010 has had as an analytical research product. Since the VDR was published more than a year ago, the emphasis of this study is on its influence on policy discussions and debates among policy-makers and donor 9 organisations, and on the debate around the governance agenda within the World Bank. The study therefore refers to evaluation terminology such as uptake and outcomes (Hovland 2007). 1) The study of uptake provides information on the direct response to the VDR 2010 by collecting data on online downloads and citations. 2) The study of outcomes provides an understanding of the direct and indirect influence of the VDR on knowledge and on policy discussions and debates (i.e. assuming that one year is too short to have policy impact). The study has been conducted in collaboration with two researchers from a local research organisation, iSEE and the main data collection activities occurred during a visit to Ha Noi from 3rd to 15th March 2011. With regard to uptake, the sources of data have been: data on sales of the VDR 2010, both in English and Vietnamese provided by the Vietnam Development Information Centre (VDIC) in Ha Noi; data from downloads of the VDR 2010 in either English or Vietnamese versions; and a web search of citations of the VDR 2010 in the following categories: donors documentation, government documentation, NGOs documentation, newspapers. The outcomes of the VDR 2010 data and information have been collected largely using semi- structured interviews with a total of 32 informants from government institutions, donor, and Vietnamese research organisations (See Figure 2 and Figure 3). The semi-structured interviews, (which are exemplified in Annex 2), examined the process that led to the production of the VDR 2010, the perceived quality of the VDR 2010, and the communication approach that was adopted for the release and dissemination of the VDR 2010. The final question explores any concerns or suggestions voiced by interviewees for future VDRs. All answers from respondents have been analysed using MaxQDA, a text analysis software, in order to guarantee, as far as possible, an objective analysis. 25   21   20   15   11   10   5   0   Foreign  informants   Vietnamese  informants   Figure 2 – Informant nationalities 10 10   9   8   7   7   6   6   4   2   1   1   1   0   Mul;lateral   Bilateral  donor   Local   Government*   NGO   Academia   Newspaper   donor   consul;ng   * The category of Government organisations includes government research institutes Figure 3 - Informant organisations Policy influence studies are more often than not qualitative by nature. A question that always accompanies these kinds of studies is whether the interviewees are a representative sample and how this issue is addressed. Qualitative studies of policy influencing are inherently limited by both the allocated time and resources administered. For the present study an effort has been made to try to organise interviews with a variety of stakeholders. Figure 2 above illustrates the main stakeholders that are represented in this study. It is the first time that the World Bank in Vietnam has undertaken this kind of qualitative influence study of a Vietnam Development Report and therefore this present study should be seen an experiment. In addition, the aim and purpose of the study is to examine the report’s influence on stakeholders who are also informants in the VDR’s development. The VDR 2010, as we will see later in this study, does not aim to influence a specific policy. Rather, it is an analytical product whose aim and purpose is far broader in policy influencing terms. Therefore, the product and its influence are as important as the process that led to its publication. Thus it is essential to understand the donors that signed up to the VDR 2010 and their opinion of this process, as well as the contribution of the VDR 2010 itself to discussion and debate among donors, government agencies and also between donors and government institutions. 4. Analysis of uptake of the VDR 2010 The VDR 2010, as well as other VDRs, are produced as public goods. The difference between the VDR 2010 and others VDRs is that an effort has been made to make this latest report as widely available as possible. 4,000 hard copies (2/3 of which were in the Vietnamese language) have 11 been distributed to government agencies, the media, participants at related events, and to 64 provincial libraries. This is a greater distribution compared to earlier VDRs.1 The data presented in this section refers specifically to uptake, which can be defined as the extent to which research and recommendations have been ‘picked up’ by others (Hovland, 2007). A caveat here is that Google searches conducted for this study have proven more difficult than expected. Therefore the data presented here represents the lower end of what can possibly be found. 4.1 Downloads and Internet traffic on World Bank websites Greater access to Internet in Vietnam has contributed to greater online access to the VDR 2010, where it can be downloaded for free. Data on the English version of VDR 2010 shows that during the period of the 15th December 2009 to 17th February 2011, the report was the most downloaded document on the site (or server) of the World Bank Vietnam office, with 1936 downloads (or 9,4% of the total downloaded during that period). Visits to the VDR 2010 page peaked, as show in Figure 4 below, a few months after the publication of the report. It is unclear, however, if the visits are by unique users or people who came back to the website several times. Figure 4 - Page views the Vietnam Development Report (English language) As highlighted above, these figures refer to the traffic on just one of the World Bank servers in Vietnam. The report was among the top 32 downloads in the period from January to December 2010 and has been uploaded as three different versions. One of the versions was downloaded 1,936 times, another 1,015 times, and a third 157 times. The full VDR in English, therefore, was downloaded a minimum of 3,108 times during that period. In addition, the Executive Summary was downloaded 606 times. Out of 20,000 documents downloaded from the World Bank’s English- language site, 19% were the VDR or parts of the report. The data for the Vietnamese version is unfortunately incomplete. The main problem is that it was not possible to track downloads of the Vietnamese version consistently from January 2010 to February 2011. The data does suggest that the Vietnamese version of the report was downloaded at least 1,522 times during March 2011. In addition, it was not possible to track pages views of the Vietnamese website until after mid-December 2010 (one year after publication of VDR 2010). Between December 2010 and March 2011, the Vietnamese site was viewed 1,279 times. However, these issues, now identified, can be easily fixed in the future. 1 Sales of the VDR 2010 are not a goal of the World Bank team. The figure is however monitored and to date the VDR 2010 has sold 78 copies in English language and 253 in Vietnamese language (source VIDC). 12 It must also be noted that the VDR 2010 can be downloaded from various World Bank and non- World Bank servers often associated with news.2 The figures provided here therefore present conservative figures of the true number of downloads and do not include all downloads from all World Bank sites. In addition, downloads from non-World Bank sites cannot be tracked. In total, VDRs are thought to have a shelf life of about five years and therefore the monitoring of sales and downloads of the VDR 2010 will continue for a few more years to come. What is missing at the moment is the possibility of gathering data on the geographic location of visitors to the VDR 2010 web page and their specific downloads (which is, for example available through Google Analytics). A second consideration refers to the changes in actual location of the internet page of the VDR on the website. Each new VDR has a dedicated page in the Current VDR section of the World Bank Vietnam website. If we look at the recently published VDR 2011 on Natural Resources, it appears on that page (Figure 5). Figure 5 - VDR 2011 web page Similarly, the VDR 2010 was published in the current VDR section of the site in 2010 and had a dedicated page for about 11 months (Figure 6). 2 For example: Vietnam Business News (here), Foreign Press Centre (here), Travel Vietnam (here), New Vietnam Corporate Company Business Registration Incorporation Setup Formation in HCM City (here), Vietnam Trade Office in the USA (here), Baomoi.com (here), Vietnam Economic Portal (here), Hanoi Times (here), VietSights (here) 13 Figure 6 - VDR 2010 dedicated web page However, with the publication of the VDR 2011, the VDR 2010 was moved to a different page in the document and report sections of the World Bank website (Figure 7). If the policy is to change the location of the VDR page after one year of publication, monitoring data may get lost and it will be difficult to retain consistency. The problem can be easily fixed by maintaining the dedicated web page and its metadata and tags. Figure 7 –VDR 2010 web page as of April 2011 14 Let’s now turn to the second tool used to analyse the uptake of the VDR 2010: citation analysis. 4.2 Citation analysis The search for citations was carried out by selecting a series of keywords both in Vietnamese and English which refer to the VDR 2010. These key words were inserted into Google’s search engine.3 A framework for the search of citation derived from Hovland (2007) (See Annex 1) was prepared at the design stage of the study. It is divided in key areas of search: • Development agencies • Government policy / strategy documents • INGOs and VNGOs • Training manuals / How to Guides • Newspapers articles • General internet citations However, despite best efforts, the citation analysis proved more difficult than expected. We were probably initially too optimistic with regard to the time required to conduct a thorough search and this framework did not seem to work satisfactorily during the first days of the search. This may be due to the fact that either the search engine returned too many results, which could not be reviewed thoroughly or that the advanced function of Google was not used fully. The decision was then made to search for citations from specific websites to find any documents that would quote or make reference to the VDR 2010 (See Annex 1). The search initially included Vietnamese websites or websites of organisations working in and on Vietnam. At a later stage it was also expanded to include international scholarly articles. In the discussion that follows, the citations refer to the period of April, 2011. The results of the citation analysis are grouped in the following categories: • Academic papers The search in Google Scholars and Google.com on the VDR 2010 (in English) returned a total of 11 citations. The VDR 2010 has been referenced in seven academic articles, one PhD dissertation, a BA dissertation, a paper by a researcher at the Vietnamese Academy of Social Sciences, and a paper published by the Centre for Development Research at the University of Bonn. • Donors documents The search in Google.com returned a total of 17 citations in papers or documents published by donors. Eight citations or links are from various World Bank websites and includes information about the launch of the VDR 2010; two blogs written by Jim Anderson, Governance Advisor at the Work Bank in Ha Noi; a report from one of the VDR 2010 dissemination workshops; and two other reports. Nine citations refer to donor reports (two written in collaboration with government institutions) and project planning and policy documents on issues of gender mainstreaming, land management, poverty reduction, social development, education, market regulations, and anti- corruption.4 The Joint Country Analysis Report to Vietnam by the Like Minded Donors Group and the United Nations has six references to the VDR 2010. A full list of development agency, INGOs, and government organisation websites that were searched can be found in Annex 1. The limited number of citation is partially corroborated by the responses of interviewees from the donors organisations interviewed for the study. They all confirmed that no news or links to the VDR 2010 3 Keywords have been: VDR 2010, Vietnam development report 2010, Vietnam Development Report 2010 – Modern Institutions, Báo cáo phát triển Việt Nam 2010 4 Organisations: UNESCO/IIEP, ILO, USIAD, UNDP, DANIDA, ADB, UNICEF, IFAD, SIDA 15 had been put on their website. In only one case did a bilateral donor organisation prepare a report about the release and content of the VDR 2010, which was in turn sent and distributed to the organisation’s headquarters. • Government policy documents While it is not expected that any specific policy and strategy documents cite the VDR 2010, 13 government websites had referred to the report. • Media and news The source for the citation analysis data presented here are the news clippings collected by the governance advisors of the World Bank, taken from newspapers (online and printed) and from government agency websites at both local and national level. A total of 92 news articles were reviewed, with 32 articles (34.78%) taken from provincial newspapers or agencies and 60 articles (65.22%) taken from national level newspaper and agencies. 61 articles (66.30%) refer to the publication of the VDR 2010 and the launch workshop. 31 other articles (33.70%) mention VDR 2010 in reference or as a quote in a report. 82 articles (89.13%) are in Vietnamese, nine articles (9.78%) are in English and one article (1.08%) is in both Vietnamese and English languages. Most articles were written directly after the presentation of the VDR 2010 at the Consultative Group meeting in December, 2009. However, there are articles also mentioning the VDR 2010, almost a year later in November 2010. 70   58   60   50   40   30   20   9   10   4   3   5   3   3   2   2   1   1   1   0   Figure 8 - Distribution of articles about the VDR 2010 Sixty newspapers/agencies have published only one article on the VDR 2010. Voice of Vietnam has published four articles and another four newspapers/agencies have published three articles in total (Figure 9). 16 70   60   60   50   40   30   20   8   10   4   1   0   1  ar;cle   2  ar;cles   3  ar;cles   4  ar;cles   Figure 9 - Number of article by newspapers/agencies about the VDR 2010 As previously mentioned, the citation analysis conducted for this study has not been as successful as could be possible. We would have liked to have conducted a more structured analysis, but this would have required, as we know now, more time and dedicated staff with considerable online expertise. Therefore the results returned and included here must be seen as being at the lower end of the real number of citations. The lesson is that the search has to be conducted more systematically and make full use of the advanced functions of search engines, using for example, the ‘inurl:’ function which allows a search of keywords only in url addresses (e.g. VDR 2010 inurl:worldbank). 5. Analysis of outcomes of the VDR 2010 The overall opinion about the VDR 2010 is very positive. All but one respondent consider the VDR 2010 of high quality and standard. The report is appreciated for the quality of its analysis, the historical perspective on governance, intuitional reforms and trends in Vietnam and the description of the specific challenges that lie ahead for the country. As one respondent declared, ‘the report is very comprehensive and touches on key development issues for Vietnam. This report stands out in providing what I think is a true and clear picture of institutional reforms in Vietnam.’ Another respondent notes that ‘the report reflects the broad political and institutional changes in Vietnam’. Despite this positive endorsement of the report, the opinion that we have collected with regard to the purpose and objective of the VDR is less unanimous. 17 Purpose  is   unclear,  19%   Purpose  is   clear,  81%   N=32 Figure 10 As shown in Figure 105, the majority of the respondents think that from reading the report the purpose and objective of the VDR 2010 is clear. The respondents state that the report’s aim is to provide information and ideas on governance and institutional reforms in Vietnam. They also highlight that the VDRs (not just this VDR 2010) are essential reading for staff from development organisations coming to work in Vietnam. Most of the respondents agree that the reports represent comprehensive background documents that synthesise World Bank knowledge, but also the knowledge of development partners on specific development issues. They also suggested that due to the topic it explores, the VDR 2010 stands out from the other publications. However, nearly 20% of respondents were not clear about its purpose and declared that the purpose of the VDR 2010 is not visibly stated in the report. While they acknowledge that this report is not an academic publication, the respondents could not find an aim outlined in the text.6 The interviewees highlighted various target groups when discussing the main audience for the VDR 2010. The majority of the respondents (23 out of 32) believe that the VDR 2010 is targeted at government institutions. Only one respondent explicitly mentioned the Communist Party of Vietnam as a target audience. Twenty-one respondents (out of 32) believed that the report is mainly for development partners (donors and NGOs). Both researchers and the academic community were cited 12 times, while the press was mentioned three times as audience for the report. These results are in line with the overall aim of the VDRs: to represent policy analysis products that do not aim at influencing a specific policy or group of policy reforms, but rather contribute to a wider discussion and debate around reforms in Vietnam. The VDR 2010 can therefore be considered successful in terms of its quality and analytical rigour. The majority of the respondents agree that the report provides a comprehensive picture of governance and institutional reforms in Vietnam. As noted by one respondent, ‘the VDR is a useful basic document covering a key area. It presents all key questions, discussions, and debate in an analytical way.’ Another respondent said that ‘the VDR 2010 presents the World Bank and development partners’ view on the state of development of the country and provides some suggestions for the future’. The rest of this section presents comments on the process that led to the publication of VDR 2010, more detailed observations on the VDR 2010 as a product, and a conclusion with further analysis of the communication process that followed the release of the report in December 2009. 5 The figures presented in this section are derived from the analysis of the semi-structured interviews. In case of responses provided by all or most interviewees it has been possible to calculate the percentages of the expressed opinions. 6 It must be noted that 20% refers to a total of 32 interviews, a small sample. Moreover the VDR 2010 is quite clear in describing the aim and purpose of the report (p.7). This suggests that the respondents could not find this section, did not read the introduction, or the sentence is not sufficiently highlighted. 18 5.1 The preparation of the VDR 2010 The topic of the VDR 2010 was decided in September 2008. Invitations to collaborate in the preparation of the report were subsequently sent to donors who were working on the issue of governance and institutional reforms in Vietnam. Interested organisations met in March 2009 to brainstorm an initial list of 12 chapters/topics. Five thematic and multi donor working groups were then organised to contribute knowledge from projects, programmes and studies and to review draft chapters. The final drafting of the report started in June 2009 and lasted until November 2009. The peer review process involved several individuals, as well as organisations. The VDR 2010 is endorsed by 14 donor agencies, which all review the content of the report before agreeing to have their name placed on the back cover of the report. In addition to the internal World Bank reviews that take place, the VDR was assessed by a review committee of Vietnamese experts, who acted in their personal capacity, and external reviewers, two of whom are based overseas, but who have lived in Vietnam and are conducting research on the country. They all collectively provided comments and feedback to the draft. The VDR 2010 was finally presented at the Consultative Group meeting in December 2009. Twenty-one interviewees (out of 32) believe that the process of consultation and collaboration, which led to the publication of the VDR 2010, was good or well managed by the governance team at the World Bank. Six respondents said that the process seemed participatory, but that ultimately it was the World Bank who decided what to write in the report. Five interviewees have not been involved in the preparation process. The majority of the respondents who appreciated the process of consultation noted the following: • The possibility to contribute with knowledge products • The openness to comments and feedback • The consultation process was better managed than previous VDRs • The sharing of information between the World Bank and participating donors • The opportunity that the working groups have provided not just in contributing to the VDR 2010, but also as a space of dialogue and lesson learning between development partners. One respondent who was involved in a few working groups commented that time constraints could have been managed better. It had been difficult to provide feedback on short notice to 20-25 pages drafts. During the interviews, only three respondents explicitly stated that the VDR 2010 is simply a World Bank product; the majority see it as speaking for the development partners’ community in a broader sense.7 Contributions and input from Vietnamese counterparts came from officials and scholars who provided comments in their personal capacity. This is an approach that has proven successful with previous VDRs and has contributed to the acceptance of the VDRs including the VDR 2010 within government institutions. Interestingly, as one respondent declared, while the VDR 2010 aims at influencing discussion and debate of institutional reforms within government institutions, the comments provided by Vietnamese experts influence the VDR 2010 itself. 5.2 The quality of the VDR 2010 The style of the VDRs has always been neither too academic nor too simplistic. One respondent cited The Economist as a style reference. The general consensus among respondents is that the style of the VDR 2010 is fit for purpose and it provides both information and suggestions in a tone that suits its audience(s). The respondents also believed that, in terms of size, it was fit for purpose and that a shorter report would have had less data and therefore be less credible. Most of the 7 It must be noted that the semi-structured interviews did not include the specific question: do you think that the VDR 2010 is a World Bank product? 19 respondents had read the executive summary and any sections of the report that were more closely linked to their areas of work, but none of the respondents had read the whole report. Most respondents considered the executive summary too long. Interestingly, most of the Vietnamese respondents think that the executive summary should be shorter and should also be a separate product as this is the traditional way to present these reports to/by government institutions in Vietnam. In terms of language, this is considered appropriate: not too academic and not too simplistic. All the respondents who could speak English have only read the English copy of the report. We could therefore not get many concrete comments on the quality of the Vietnamese translation, though one Vietnamese respondent said that the Vietnamese translation was very good and has also helped to communicate the message of the VDR 2010 at sub-national level. However, another respondent mentioned that in the Vietnamese version some technical terms do not seem to have been translated correctly. This is an area that can be explored for future versions to maintain a high quality for the Vietnamese version. During the interviews it was interesting to see the link between the issue of the purpose of the VDR 2010 and the style of the report. As mentioned earlier on, the purpose of the report was noted as unclear by approximately 20% of the interviewees. This is reflected as a point of discussion, which came up in almost all the interviews undertaken: policy recommendations. Some respondents said that there is an expectation that reports like the VDR 2010 should end with a list of policy recommendations. This group of respondents said that such a list would contribute to help clarify the purpose of the report and that without recommendations there is uncertainty as to what to do with the report and the knowledge that is embedded in it. This was an issue discussed in all the interviews. The lack of a list of explicit policy recommendations is a choice made for the VDR 2010 by the authors of the report, because the topic of the report is quite broad and touches on sensitive issues. Suggestions for reforms on key institutional reform agenda are therefore included in the overall text. Explicit policy recommendations would cover too broad an area and result in a confused message on next steps. Twenty-eight respondents (out of a total of 32) agree that implicit recommendations described as suggestions and included in the main text have been the right approach for the VDR 2010 and have contributed to the positive response of the report. More explicit policy objectives and recommendations would make the VDR 2010 a very different document and would alter the type of contributions and the negotiation required with development partners to partake as well as the participation of Vietnamese reviewers in their personal capacity. One respondent said that while agreeing that explicit policy recommendations may not be practical, the tone of the report should be more political and address political obstacles to the intuitional reforms more directly. One additional issue that was mentioned in a few interviews needs to be noted here. Five respondents have mentioned that the credibility of the report is also linked to the credibility of the data that informs the report. The VDR 2010 draws a large part of its data from the governance component of the Vietnam Household Living Standards Survey conducted in 2008 implemented by the General Statistics Office of Vietnam and which the government has yet to make public. 5.3 The communication strategy of the VDR 2010 Among all interviewees, twenty six believe that the VDR 2010 communication strategy has been well managed. As well as the presentation of the report at the Consultative Group meeting in December 2009, the World Bank organised an official launch and two more workshops in Ha Noi. What has been different compared to previous VDRs and was highly appreciated, are the seven public dissemination workshops that accompanied the report and which were held during 2010.8 These workshops were organised in collaboration with three government institutions, the Institute of Legislative Studies at the National Assembly, the Vietnam Fatherland Front and the Training Centre for Elected Representatives (TCER). A local research NGO, the Centre for Development Studies & Community Support (CECODES), was also involved and has organised three 8 One in each of the following provinces: Ha Noi, Hue, Hai Phong, Da Nang, Dien Bien. Two in Ho Chi Minh City. 20 workshops. In total, some 500 Vietnamese participated in these workshops. The three workshops organised by TCER had nearly 200 participants coming from 52 provinces/cities, many of whom were National Assembly deputies. This approach of the dissemination workshops conducted both at central and local level has also proved successful for other projects in other donors organisations.9 The VDR 2010 workshops allowed government counterparts at national and sub-national level to prepare comments and make public their views on the content of the VDR. In addition, the VDR 2010 was used as a reference by members of the governance team of the World Bank when they participated and presented at seminars and workshops organised by different donors working on governance and institutional reforms in Vietnam. One respondent highlights at least twenty meetings that have been organised within various offices of the Communist Party of Vietnam on the VDR 2010. 5.4 Contributions of the VDR 2010 The VDR 2010 is considered a timely report. It touches on the key topic of the future of Vietnam and the development of Modern Institutions. We have seen in previous sections that a small number of respondents were unsure about the extent of the report’s influence, especially on government counterparts, due to their perception that the purpose of the VDR 2010 was unclear. However, almost all respondents could give examples of the VDR being cited in discussion or being used in the design of a project or programme: Design of the Social Audit Pilot Project currently implemented in four provinces of Vietnam in collaboration with Ministry of Planning and Investments and the Central Institute for Economic Management. UNICEF A budgeting study that looked at budget allocation in the health and education sectors in three provinces. The report refers to the VDR 2010 as source of background information. A study on Child Rights Monitoring mechanisms and the role of different local institutions conducted by Children Legal Centre of the United Kingdom An internal discussion about the discrepancy between the national annual growth rate of 7% and the 12% rates from the Provincial GDP index. Discussion about the difference of statistical data and approaches for data collection and the ongoing debate about the need to increase the General Institute for Statistical Office capacity and achieve greater evidence-based policy-making in Legislative Vietnam. Studies at the National Several reports prepared by Institute for Legislative Studies in 2010 used the Assembly VDR 2010 as a background paper though they do not quote it directly Before the 8th National Assembly session, the Institute for Legislative Studies received between 1500-1700 questions for clarification from National Assembly members on various issues. For some of the answers the VDR 2010 was used as a reference. Swiss Development SDC used the VDR 2010 as a reference for its yearly country analysis, which Cooperation was then shared with its headquarters. It also served to inform staff and HQ. Canadian International Development The VDR 2010 was used as a reference for the planning exercise on the next Agency five years of Canadian support to the Government of Vietnam. 9 For example Acuña-Alfaro, J. (2009) Reforming Public Administration in Vietnam: Current Situation and Recommendations, Ha Noi: UNDP. 21 Vietnam Fatherland The VDR contributed to some discussions and meetings and was also used as Front source and reference. The VDR 2010 workshops helped the discussion on grassroots democracy, data sources, and particularly on conflict of interests of elected representatives. Training Centre During the 8th session of the National Assembly, several members mentioned for Elected the VDR 2010 when discussing decentralisation and devolution, conflict of Representatives interest (linked to management of natural resources), roles of central government vs. local government, and the shortcomings of administrative reforms in Vietnam. However, there are no public records of these discussions. Government The VDR 2010 is expected to be used as a reference and background in the Inspectorate discussion starting in March this year on the accountability decree. Fulbright School in Ho Chi Minh The VDR is used as a textbook for two courses: Law and Public Policy and City Public Governance, which involve 60 public officials. The VDR input into the definition of the Anti-Corruption Training Programme that AusAID was developed for staff on Communist Party Inspectorate, GIs, and the office of Steering Committee for Anti-Corruption. CECODES The VDR 2010 was used to inform the Public Administration Performance Index questionnaire on issues of participation right and land use rights. UN’s One Plan for 2012-2016 United Nations Reference for the discussion on how the UN could support Vietnam to strengthen local governance in the coming five years. Reference to the Joint Country Analysis conducted by the fourteen Like Minded Group of Donors. Finland Report on the VDR 2010 sent and distributed in the Ministry for Foreign Affairs in Helsinki. Copies of the VDR 2010 were sent and distributed in the Ministry for Foreign Denmark Affairs in Copenhagen. Some of the example listed in the table could be studied in further depth and presented as stories of change on the influence of the VDR 2010. 6. Conclusions and recommendations The aim of the VDR 2010 is to provide information and ideas that can contribute to the discussion around governance and institutional reforms in Vietnam. The results from the interviews for this influence assessment study confirm that the VDR 2010 is so far achieving this aim. While it is difficult to assess the use and uptake of the VDR 2010 in discussions within the Communist Party and government institutions, we need to remember that the reports itself is influenced by the Vietnamese reviewers who participated in the preparation process. In this respect, the VDR 2010 represents donor views on not only the past, but also the present and future of modern institutions in Vietnam. No piece of research or report can satisfy all audiences or meet the expectation of all audiences. However, the VDR 2010 is considered as a key reference for anybody working or researching 22 these issues in Vietnam. It would be much more difficult to gather all the information and data from various sources if these were not combined in a publication such as the VDR 2010. More importantly the respondents to this study appreciate the process that led to the development of the VDR 2010, as well as the communication approach adopted for its dissemination. The majority of the respondents are satisfied with the VDR 2010 as a product and a process. The main additional suggestions mentioned during the interviews are: Two respondent mentioned that it was now time for a joint A joint report VDR to be written by donors and the Government of Viet Nam While most of the respondents agree that explicit policy recommendations would not be useful, six respondents Additional policy briefs suggest that a few policy briefs could be produced to address key issues highlighted in the VDR and used to communicate specific policy changes or actions. Eight respondents suggest having a shorter executive summary (also printed as a separate product). This would A shorter executive summary increase the chances that top-level leaders would access the VDR. One respondent suggested that the VDR should be presented Disseminate the report among at workshops organised in universities and government Vietnamese research institutions research organisations. The present study is concerned with the influence that the VDR 2010 has had as an analytical research product. It is useful to briefly list here the areas of monitoring and evaluation of the influence of policy research described by Ingie Hovland (2007) and see how these are linked to the VDR 2010: Area of monitoring Tools VDR 2010 Comments and evaluation Is an output that Strategy and direction Theory of Change describes change in of the policy research Vietnam. Good management Research management of the preparation Project Management approach and communication Documentation of the process could benefit from an After Action Research outputs Peer review Done Review and also comments by members of working groups and reviewers Documentation could Citation Analysis; benefit from uptake logs Research uptake Semi structured The current study linked to the VDR web interviews site (specific suggestion below) Documentation could Semi structured Research outcomes The current study benefit from in depth interviews case studies selected 23 from the ones listed in the present report. Case studies, Stories As part of overall impact Research impact of Change, Most Later on and influence analysis of Significant Change GAPAP Our specific recommendations are: • Describe the Theory of Change that includes the VDR 2010 • Keep the VDR as a joint donor exercise • Keep the working group approach for consultation and involvement of donors in the preparation of the VDR. • Keep the review committee’s approach with Vietnamese experts providing feedback in their personal capacity. • Maintain the style of including implicit policy suggestions rather than explicit policy recommendations. • Continue the dissemination workshops approach and include and organise a few more specifically for the Vietnamese research community • The area that could be improved the most is the monitoring of the uptake of the report. For example, bit.ly links and embed tracking with Google Analytics in the VDR web site. The benefits are that it could help to better understand how and why people are coming to the VDR website. These tools can also help to monitor uptake of publications on different platforms. Bit.ly, for examples, has reports based on number of clicks and the geographic origin of clicks. Both are free services and are relatively easy to set up. • Feedback forms could be uploaded to the VDR 2010 website and could appear whenever the VDR is downloaded. Additional recommendations include using a simple ‘impact log’ in the form of an email address which could be set up to encourage staff and counterparts to send stories or other instances of impact and uptake of the VDR. The main drawback is that an impact log cannot be considered systematic or scientifically rigorous, but it can be a good source of qualitative information to back up certain influencing claims. Additionally, when using an email address information is locked there and someone has to categorise, store and share that information. A Google Alert should be set up as soon as the VDR is published. All these services are free of charge. • The dissemination of VDR would benefit from donors, who participated in the production of the VDR 2010, placing a link to the VDR web site on their own websites. This is justified by the fact that they contribute both knowledge products and staff time to the preparation of the report. American-born Canadian writer and activist Jane Jacobs in her book ‘The Nature of Economies’ (2000), highlights the complexity and unpredictability of development describing it as ‘an open ended process, a qualitative change that can’t be usefully thought of as a line, or even a collection of open-ended lines. It operates as a web of interdependent co-developments,’ (p. 19). Development is therefore an incremental process that with time increases in complexity. Similarly, the Executive Summary of the VDR 2010 mentions that ‘the shifting landscape of devolution and accountability is neither linear nor predetermined. Indeed, the Vietnamese approach to reform, searching step-by-step for solutions to society’s problems, is one of experimentation. And experimentation, pushing boundaries in one way or another, continues to occur.’ (p. xv). The development of Vietnam fits Jacobs’ description. The two priority areas since the Doi Moi reform in 1986 have been firstly concerned with food security and then poverty reduction though a 24 socialist-oriented market reform. In the meantime, the complexity of development has increased and for Vietnam to remain a middle income country, reforms will need to focus on the development of modern institutions. The next decade, as noted by one of the respondents of this study, will be the decade of governance and institutional reforms. The quality of the analysis of the VDR 2010 is not only a timely assessment of the current situation, but will provide suggestions and ideas for the coming years. 25 References   Acuña-Alfaro, J. (2009) Reforming Public Administration in Vietnam: Current Situation and Recommendations, Ha Noi: UNDP. Hovland, I. (2007) Making a Difference: M&E of Policy Research, Working Paper 281, London: ODI. Jacobs, J. (2000) The Nature of Economies, New York: Modern Library Jones, H. (2011) ‘A guide to monitoring and evaluating policy influence’, ODI Background Notes, February 2011. Available at: http://www.odi.org.uk/resources/details.asp?id=5252&title=monitoring- evaluation-me-policy-influence Jones, N. with Villar, E. (2008) ‘Situating children in international development policy: challenges involved in successful evidence-informed policy influencing’ in Evidence and Policy, vol4, no.1, 2008 p.53-73 Keck, M. and Sikkink, K. (1998) Activists beyond borders: advocacy networks in international politics, Ithaca, NY: Cornell University Press Nielson, S. (2001) Knowledge Utilization and Public Policy Processes: A Literature Review, IDRC supported research and its influence on public policy, IDRC Evaluation Unit http://www.idrc.ca/uploads/user-S/105223752702-literature_review_final11_by_Neilson.doc Start, D. and Hovland, I. (2004) Tools for policy impact: a handbook for researchers, London: ODI http://www.odi.org.uk/resources/download/156.pdf 26 Annex 1 – Uptake data Citation Analysis on VDR Development agencies Agency Document Title Eng Vn Eng/VN Publication Date Web link Cited in text Cited in reference list Government policy /strategy documents Ministry Agency Policy document Eng VN Eng/VN Publication Date Web link Cited in text Cited in reference list INGOs and VNGOs NGOs Document title Eng VN Eng/VN Publication Date Web link Cited in text Cited in reference list Training manuals / How to Guides organisations Document title Eng 27 VN Eng/VN Publication Date Web link Cited in text Cited in reference list Newspapers articles Newspaper Article title Eng VN Eng/VN Publication Date Web link Cited in text Dedicated article (how much space) General internet citation Key words Number of links Web sites search Donors websites: − ADB − AECID − Ausaid − Belgium Technical Assistance − CIDA − DANIDA − DFID − EC − Finland − GIZ − Jica − Luxembourg − SDC − SIDA − UsAID NGOs working in Viet Nam web sites: − Actionaid − Asia Foundation 28 − CARE − Church World Service − East Meets West Foundation − Ford Foundation − Friedrich Ebert Stiftung − Helvetas − Oxfam Great Britain − Oxfam Hong Kong − Oxfam Quebec − Oxfam Solidarity Belgium − Plan International − Save the children − Save the Children Australia − Save the Children Fund United Kingdom − Save the Children Japan − Save the Children Sweden − Save the Children USA − Swiss Foundation for Technical Cooperation − VECO Viet Nam − Voluntary Service Overseas − VUFO-NGO resource centre − World Vision International Government web sites: − Government of Viet Nam − Communist Party of Viet Nam − Ministry of Planning and Investment − Ministry of Justice − Ministry of Home Affairs 29 Annex 2 – Semi structured interview Name: Organisation: How long have you been working for your organisation? What is your role? What is your overall opinion / impression about the VDR 2010? 1) Identification of specific areas/sectors in which the VDR has had influence What do you think is the purpose of the VDR 2010? Who do you think is the main audience of the VDR 2010? How did you use the information and data contained in the VDR? Who in your organisation normally reads this kind of report/publications? Did the VDR lead to new discussions within your organisation/institutions? (Example) Did the VDR suggest new activities, projects, way of conducting research, policy areas/discussion? (Example) 2) Perception of the process that led to the VDR Were you part of a working group that led to the VDR? What is your opinion on the consultative process that led to the VDR 2010. 3) Perception of the quality of the VDR 2010? How do you find the structure? Length? Data presentation? How do you find the language? 4) Perception of the VDR communication strategy How did you find out about the VDR? How was the VDR disseminated in your organisation? 5) Suggestions for the VDR What could be done to improve the preparation and dissemination of these types of studies?