Evaluating Katine

Conflicting views of the likely success of the Katine project

The Katine Community Partnerships Project was scheduled to end in September 2010. In early 2010 AMREF submitted a proposal for a fourth year extension, up to the end of September 2011. By then £3,249,994 will have been spent on Katine, if all goes well. This is equivalent to £108 per person in Katine, assuming a population of 30,000 people.

At the end of the same proposal AMREF make the bold claim that: By the end of a fourth year in Katine, we will aim to have met all our objectives of improved quality of life for the people of Katine.

The same proposal begins with an even more ambitious expectation by “community representatives and local government at the midterm review workshop in Soroti” in September 2009 that “In  2014,  Katine will  be  an  active,   empowered  community  taking  responsibility  for  their  development with decent health, education, food security and able to sustain it with the local government”

Jo Confino’s final comment in the online debate on the extension proposal is optimistic about the Guardian’s role but more pessimistic about the developmental impact of the KCPP: “So at the end of the year four, Guardian will be able to congratulate itself with fulfilling its media objectives. As for development objectives, I am afraid that I feel rather pessimistic.”

Richard Kavuma’s recent analysis[1] of the Katine sub-county’s capacity to continue to fund KCPP activities post-2011 is also downbeat: “Given the huge discrepancy between the needs of local governments and their capacity to finance those needs, the possibility that Soroti district and Katine sub-county officers will fill the void that will be left by AMREF seems overly optimistic.” He notes that the budget for Katine sub-county in 2009 was UShs 32 million (£42,106), equivalent to about 5% of the annual cost of the KCPP project.

How can anyone tell?

AMREF’s extension proposal has made no budget provision for any evaluation activities in the last year of the project.

There is £3,200 for monitoring and evaluation “Capacity Building”, possibly for elected officials and members in community groups and staff in government offices at sub-country and district level. It is unlikely that these groups will be able to take responsibility for any summary evaluation of the KCPP, given the wider financial constraints on council spending highlighted by Richard Kavuma.

There is also £5,221 budget line for “Documentation”. This may relate to the section of the exit/sustainability plan, where it is stated that: “Analysis of the Katine model, documenting and sharing best practice within Soroti and advocating for its  scale  up  and/or  replication  at  national  level  by  government  and  bilateral  donors  in  Uganda, especially  in  the context of  the Uganda government policy on Public Private Partnership  for Health (PPPH).” (Italics added)

Ideally documentation used for advocacy purposes would be evidence based. But where will that evidence come from? The amounts quoted above stand in pale comparison to the Monitoring and Evaluation budget line for the first year of the project (£17,977), where the biggest cost was the baseline surveys.

Although there is no specific commitment in the extension proposal it is possible that AMREF will continue to provide six-monthly and annual progress reports. By itself this reporting is unlikely to provide sufficient independent and appropriate evidence to resolve the differing claims made above. Resolving them does matter, both to how advocacy work is done within Uganda and how wider lessons are learned from the project by international audiences following the project through the Guardian website

The situation on the Guardian side looks equally unimpressive. There is no budget available for further external evaluation of the project during the last two years of the project (2010-2011). In contrast to the project itself, the Guardian is going through a process of budget cutting, associated with the wider difficulties of the Guardian group[2].

What could be done?

The project has two major assets that could be helpful

Firstly, the KCPP has probably the largest audience of interested bystanders of any single development project that has been seen for a long time. They have been attracted by what must be one of the most comprehensive project websites that has ever been developed. In additional to information provided by AMREF there are also numerous articles both about the details of Katine people’s lives, the political and economic issues affecting them and how these relate to the wider Ugandan context. There is a large amount of information available, and large audience of potentially interested users of that information.

Secondly, the Guardian has its own resident independent Ugandan journalists who have been the produced much of the information on the website. Hopefully they will continue to cover the project until it comes to an end. During this remaining time it would be possible for their work to take on more of an evaluation function. This could be done through means of inquiry they are already familiar with: a series of in-depth interviews with a wide variety of people, both at the end of the project, and then with the same people at least one year later. Ideally the people interviewed would be from all four of these groups: (a) Households (men, women, children), (b) Community groups (e.g VHTs, PTAs, WSCs,Farmers Groups, VSLAs etc), (c) Staff managing health and education services in Katine, (d) District and sub-county officials.

While it is important to know what impact the project activities have had by the end of the project, the more important question is the sustainability of what has been achieved. With this concern in mind the journalists could focus on obtaining (testable) predictions from the people they interview about what the situation in Katine will be like in a year’s time, after the project has finished. These predictions could be about: (a) the state of schools and health services, (b) the functioning of community groups, (c) the welfare of households, Ideally these predictions would specific rather than general, and include some explanations. For example, about the number of water sources that will still functioning, the number of functioning VSLAs, the number of women coming for immunisation and ANC at the health centres, etc. While many people may not be able to predict actual numbers it is likely they could be willing to predict the direction of change,  i.e. increase, decrease or no change. In a year’s time the Guardian could fund a return visit by the journalists to find out what has actually happened, and why.  A lot could be learned from such an exercise.

Some questions about the current impact of the project activities could also be explored. This could be done through comparison questions of what has been provided, and with what has not been provided. For example, individuals could be asked:

• “If you were given UShs 300,000 at the beginning of the AMREF project in 2008, how would you have used it?
  • What about the other members of your household, if they were also given the same amount?
• “From what you know about what AMREF has done here since 2008 would you have preferred that AMREF kept that money and used it for community development activities here in Katine?

If the Guardian was able to follow this journalist-led-evaluation path they may be able to test a hypothesis of my own: that in some circumstances it may be more cost-effective for donors to contract independent journalists than to hire external evaluators. So far the Guardian Ugandan journalists have produced much more informative reporting on the project than I have produced as the external evaluator. Their reports have probably been much more widely read and have provoked more responses than my own. And they have cost less.

Crowdsourcing

This was an idea of interest to the Guardian at the beginning of the project. Via the Katine website, it has been applied to problems relating to material supplies (e.g. solar panels to provide electricity where the mains system is not working) and also to questions of project design (with less clear success).

Given the volume of data collected on the Katine project and community, and the continued availability of the Guardian journalists on location, it would seem to make sense to try to crowd source the names of some potentially interested parties who could make best use of it all, and in the process synthesise some final conclusions. I am thinking of research and teaching institutions as the most likely candidates.

[2] http://www.telegraph.co.uk/finance/newsbysector/mediatechnologyandtelecoms/6545303/Guardian-News-and-Media-to-make-more-than-100-new-job-cuts.html