Mid term evaluation report (2011-2014) – HIV material and training to the churches in Ethiopia

Background

The Bible Society of Ethiopia (BSE) started the Good Samaritan HIV program in 2005. 2011-2015 is the second phase of the project. This evaluation was initiated by the Bible Society of Ethiopia and its partner the Norwegian Bible Society. The evaluation aims to identify the program’s contribution for the reduction of HIV incidence and prevalence in the program areas and to see the whether the program achieved its predetermined outcomes.

Purpose/objective

The objective of the evaluation was to assess whether the project achieved its predetermined objectives efficiently and effectively; to see whether the project was relevant to beneficiaries and other stakeholders; and assess whether the project achieved the impact/outcome and the sustainability of the results. 

Methodology

This evaluation was conducted using the standard methods.  The sample size was fixed purposely but the numbers of interviewees were clustered into different groups such as youth, women, congregation members, PLHIV, etc. After clustering, sampled respondents were randomly selected from the list provided to the evaluation team.  Data were collected using both qualitative and quantitative tools.  Sampled beneficiaries were interviewed using the structured questionnaire while stakeholder were asked using question guideline prepared for key informant interview and focus group discussions. Different documents, reports and secondary data were also referred. The results mentioned in the physical reports were triangulated by individual interview, key informant interview and FGDs.  The quantitative data were entered to computer and analysed using SPSS V20 software.

Key findings

1. In general the performance of the project can be said very good as almost all activities are accomplished above the plan.
2. The project was well designed but lacked logframe and documented baseline data
3. The project was highly visible among the community members and outside them
4. The project has contributed to behavioural change in the project areas.
5. The IEC/BCC materials were effective in conveying the HIV/AIDS prevention messages not only to beneficiaries but also other peoples who were not directly addressed by the project.
6. The project encouraged many people to take VCT and sexual education, but due to absence of data management system it is not easy to track this achievement.
7. The project played important role in reducing stigma and discrimination. 
8. The project was able to renew church’s responsibilities in relation to playing their paramount role for the elimination of HIV and strengthening care and support services to these people.
9. With respect to financial management, BSE has well equipped accountant and there is inbuilt financial management system. 
10. BSE does not have a separate M&E unit, this activity is covered by the project manager.
11. The project has no phase-out strategy and plan.
12. Strong project partnership and cooperation with government and other stakeholders.
13. In general; the evaluation results indicate that; The project was relevant to the needs of beneficiaries, appropriate to the government organization; The project was effective in addressing the desired stakeholders and beneficiaries; The project was efficient as it addresses many people with very low amount of money; The project has brought positive impacts and achieved its outcomes on beneficiaries and other community members particularly in respect of abstain among youth, reducing stigma and discrimination and renew church responsibility; Churches started to own the project in order to make the results sustainable.

Recommendations

1. Should strengthen to receive feedback from partners and other stakeholders involved in the Program both at the time of their involvement and at a later date (for example, a year after their involvement) to assess and collect information on the Program’s impact
2. It is paramount to conduct a baseline before designing a Program as done for community health need assessment and planning
3. The M&E system should be strengthened and the Program teams should be capacitated in monitoring and evaluation to improve the departments’ performance
4. More support should be given at branch offices level
5. Should strengthen its support on capacity building and experience sharing at all levels
6. Should allocate sufficient fund or should utilize the budget in a way it will bring change instead of dispersing in different regions here and there.
7. Program proposal should be prepared based on one of various methods that show indicators, of verification along with the achievable results on the target population.
8. The Printed materials should be updated and re-written in different languages other than Amharic and Oromifa
9. In order to contextualize the messages the DVD drama should be done by Ethiopian young artists if possible in different languages
10. Should prepare Program proposal with participation of potential beneficiaries and should made regular joint review meetings
11. Should carefully design sustainability means and phase out strategy. 
12. The beneficiaries and stakeholders should hear the independent evaluation report and their comments should be incorporated so that everyone will learn from previous problems and own the phased out Program

Comments from the organisation

The Bible Society of Ethiopia has provided a separate management letter responding to recommendations and findings. The Good Samaritan HIV Program in Ethiopia is implemented in a number of areas all across Ethiopia. It was decided that the evaluation should be limited to the Dire Dawa region in order to possibly do some findings at impact level through in depth interviews. The evaluation team carried out extensive field work including in depth interviews, but did not have time to make use of the in depth interviews carried out and analyse the findings from these in particular. They do however form the basis for the overall findings.

Comment on key finding 2: The project does have a logframe showing results and indicators. It does however lack assumptions and input. The project uses questionnaires. A baseline using these questionnaires was carried out initially.

Comment on key finding 6: The reason why they lack data on the VCT is not absence of data management system. They have good systems for data collection and analysis. The reason is that the program itself does not provide VCT, however they encourage it, link people up with relevant institutions, and invite health workers to the seminars. It is correct that this makes it difficult to track the results.

Comment on key finding 11: The project developed a phase-out strategy and plan. They succeeded in having churches carry more of the financial burden and pay for workshops. However, this has not resulted in reduced donor dependence as work has simply started up in new areas and new groups.