49424

                                        Core Course:
   Budgeting Processes and the Analysis and Management of Public
                                       Expenditures

Ray C. Rist                  WBI Evaluation Unit World Bank                        Natalie Deveaux
Advisor                                                                           Evaluation Analyst
(202) 458-5625                                                                      (202) 473-1399


Number 13                                                                            August 1998


Campos, Ed
(Notation: The World Bank Institute was formerly called the Economic Development Institute
(EDI), as reflected in some text)

A Core Course on budgeting processes was presented by the Economic Development
Institute (EDI), in Washington, D.C., from June 15th to the 23rd, 1998. A total of 39
participants from 25 countries attended the course. Usable end of course assessment forms
were returned by 28 respondents, 72 percent of all participants. The objective of the course
was to provide a forum in which to discuss the issue of public expenditure management
system reform. The methods of instruction included lectures, group exercises, and case
studies.

The EDI Evaluation Unit (EDIES) conducted an evaluation using an end-of-course
questionnaire. The questionnaire consisted of two sections. The first was a survey of
participants' opinions of the course objectives and delivery. The second section attempted to
gauge the increase of knowledge gained from the course, using a set of self-assessment
questions. This evaluation pioneered a new self assessment device: the introduction of a
control question to increase the validity of the questionnaire. Both sections of the evaluation
used a 5-point Likert type scale that ranged from 1=minimum to 5=maximum.

Respondents had varied educational levels and professional backgrounds. Twenty-one of 31
had a Masters or equivalent. There were 4 respondents with a Bachelors or equivalent and 3
with a Doctorate or equivalent. Twenty-six of 31 worked in the public sector, although the
number of years experience in the field of budgeting, finance and/or development varied.
Nineteen of all the respondents had less than 10 years experience and 12 had over 10 years
relevant work experience. No information on gender was collected. Results from the
evaluation are summarized below.



     l  The questions asking the respondents for their opinion of the course fell into 2 major
        categories. The first category addressed the objectives of the course, and asked
        questions about whether the respondents had learned new information, concepts and
        policies. There were three such questions asked, and the mean scores ranged from
        3.77 - 3.94 out of 5. The second category addressed the design and delivery of the
        course. Questions asked were related to course materials, faculty communication,
        and level of interest, among other topics. There were 7 questions asked under this
        category, and the mean scores ranged from 3.81 to 4.45.


     l  An overall question asking respondents if the course was a worthwhile use of their
        time received a mean score of 4.52, which was the highest mean score of the
        evaluation.


     l  Due to small numbers of private sector respondents as well as respondents with
        either a Bachelors or a Doctorate degree, analysis by sector or academic group was

 problematic. However, comparisons were made between those respondents with less
 than 10 years experience and those with greater than 10 years experience. No
 significant differences were found between the two groups in terms of satisfaction with
 the course or knowledge gained.


lThe highest and lowest scores for each item of this course were favorable. No item in
 this evaluation received a rating of 1, the lowest possible score. Moreover, every item
 received at least one 5, the highest possible score.


lFor the first time in EDIRP, a control question was used in the self-assessment
 portion of the evaluation. This was done to eliminate influencing factors other than the
 teaching of the course. To control for these non-teaching influences, a question was
 asked which had not been taught in the course. The pre-course mean score for this
 control question was 3.03, and the post-course mean score was 3.70, which is a
 perceived change in knowledge of .67. This number, which represents the (perceived)
 knowledge gain from non-teaching influences, was then subtracted from the pre/post
 change values of the non-control questions. The result was a net pre/post change
 score which relates more precisely to the teaching of the course. The range of net
 pre/post change scores was .10 to .70.


lThe range of the pre/post change scores may point out areas of relative strength and
 weakness of the course. The lowest net change score was .10. This was given for the
 item covering "general principles that govern the choice of public or private delivery of
 services". The highest net change score was .70. This was for "the role of
 performance measurement in encouraging accountability of public servants".


lOne limitation of this evaluation is that the pre/post data are based on self-evaluation
 and therefore measure only perceived knowledge gains. A cognitive-based test done
 before and after the course would better measure actual learning gains. In addition,
 this evaluation may not accurately portray the opinions of all participants as some did
 not complete a questionnaire. Therefore, it is important to encourage all participants
 to provide feedback.