Evaluation of the PDI-Bol institutional development project (2007 – 2011 period)

Background:
The PDI-Bol institutional development project, carried out since the 2007 – 2011 period through FES, has proposed to improve the quality of education in the units (schools)
and transforming the Educational Units (EU) into centers of pedagogical innovation offering quality education, responding to basic learning needs, and in which children and adolescents develop productive and competitive capacities.
To date, the PDI-Bol, as a project in the area of education, has proposed carrying out the following strategies, or five fundamental pillars, for achieving the overarching goal of “Quality education through Christian principles and values”:
• Participation of the Educational Community
• Professional development of teachers
• Specializing and outfitting classrooms
• Updating plans and programs
• Monitoring and evaluation

Purpose/objective:
At the request of FES for the five-year evaluation of the PDI – Bol 10585 project, with the aim of verifying the results obtained according to organizational goals; reflecting institutional learning at the end of the fifth year, to develop and improve educational practices.

Methodology:
Interviews
Field observations
Check list
Review of documents
(LQAS) Lot Quality Assuring Sampling
ISO/IWA-2 International Workshop Agreement 2 tool

Key findings and recommendations:
(Due to an extensive list in the main report, the selection presented here is compressed. Please refer to pages 89-92 in the report for a complete overview.)
1. Strategic Focus
Officials responsible for the PDI-Bol project and those who make up the FES framework entity have the task of seeking expert advice. These experts should help them structure and better formulate their ideas to translate them into clear and precise systems of planning, implementation, evaluation, monitoring and control.
2. Focus on Processes and Document Structure
The Five-Year Strategic Institutional Development Project should be presented in a document approved by FES, CEN and the funder.
All Requests for Incorporation and Project Contracts of the fourteen Educational Units under the PDI-Bol agreement must be completed and/or updated. At present, contract errors legally invalidate the documents.
3. Focus on Resource Management
Actions should be taken to ensure the ready availability of accounting ledgers that support all movements and allow them to be traced.
4. Focus on Service Provision or Project Implementation
Pillar 1. Equipment and specialization of classrooms
In view of a lack of clear procedures for the internal control of Educational Units’ infrastructure and previously reported important findings of deviant situations, to which one could not evidence a clear response, one should consider at least the following elements of internal control:
1. An approved procedure for Purchase Planning and Management.
2. Procedure for codification and for the realization of periodic inventories in EUs.
3. Procedure for the control of equipment use.
4. Procedures for the control of equipment that requires repair or that should be retired.
Pillar 2. Participation in the Educational Community
Parents and the management team in general need their structure and operational procedures defined, considering the importance of their organization for EU planning, implementation, monitoring and improvement.
Pillar 3. Teacher Training Program
Among essential tasks for any program’s success, in this case an educational program aimed at teachers, one should consider project planning, identification, preparation, evaluation, monitoring and control. This makes it possible for the project to develop in a strategic planning framework that allows project to order, drive and orient actions toward the fulfillment of objectives.
Pillar 4. Curricular Innovation
Any curricular proposal should have important support in matters of education management, training and monitoring, In addition, one should carry out pilots and controlled tests of any didactic material to be introduced into the Educational Units, observing the materials’ long-term potential.
Pillar 5. Monitoring and Evaluation
FES has a pressing need to promote or institutionalize four basic elements of planning: long-term perspective or vision, coordination, evaluation and strategic harmonization.
Although one has defined the instances responsible for monitoring, analyzing and improving PDI-Bol activities, be it for lack of assigned resources (CEN) or lack of clarity with regard to the reach of responsibilities (National Management – FES Accountant), now that 90% of project period has passed, there are still no procedures or clear responsibility for this process; therefore, it is recommended that one formally create a hierarchy of authorities with responsibilities.

Comments from the organisation, if any:
Supplementary Information from the PDI Team on the External Evaluation:
The external evaluation was carried out according to the agreed-upon terms of reference for the evaluation. All formalities were complied with during its execution. The multidisciplinary consulting proposal contributed to satisfactory work being performed.
The assessment states that the teachers who have been trained by the Bolivian Evangelical University have better performance than the teachers who have not been trained; (according to page 51 part 3 of the evaluation report). This shows us that all the effort and investment in favor of developing the professional skills of teachers has been worthwhile, but there is certainly much to be done yet, because professional development is continuous. The evaluation also states that the students of teachers who have been trained achieve better results than students of teachers who have not been trained, and the satisfaction of clients (students, parents) is positive. These last two results are momentous for the PDI team because the beneficiaries are satisfied and the outcome of the students has improved. In other words, the project is on the road to achieving its most important challenge.
The evaluation report mentioned, very briefly, that management teams have been formed in each educational unit. Our team highlights this achievement because the base of the PDI project is the participation of the educational community through management teams. This result is being achieved. It is also true that we must regulate the participation through regulations and continuous training processes to strengthen the participation of the educational community and to engage in the search of a higher quality for the education of children, young men and women. Another very important aspect is that the evaluation mentions that the classrooms are equipped. That was another very tough challenge, because we believe that it is very difficult to achieve quality in education without giving adequate material conditions so that teachers and students develop a better teaching-learning process. The classrooms are now equipped with audio-visual, didactic and bibliographic material. That will definitely have an impact on the quality of education. It is true that teachers are still not giving the best use to the equipment purchased, but this is a process and a challenge for the future.
We believe it is appropriate to clarify that the program of professional development of teachers, executed by the Bolivian Evangelical University, succeeded in that 35 per cent of professors and 33 per cent of Directors fully completed the modules (according to page 27 of part two). Approximately another 30% have achieved good performance. These results are supported by the statistical table provided by the Bolivian Evangelical University. The table is signed by the head of the UEB’s Education Sciences Department, and transcribed in the lines below:
Students of the PDPD UEB Program in 2010:
Total teachers admitted to the program:      476
Total teachers who have completed the program:     165
Total teachers who have made good progress but have not completed the program 135
Total teachers that achieved less than 50 per cent or deserted   125
Total Directors that joined the program:      12
Total directors who have completed the program     4
Total directors who have made good progress but have not completed the program 2
Total directors who achieved less than 50 per cent or deserted   3
Total directors completing the diploma      3
Massive Seminars         750
About massive seminars we must indicate that they began to run in the 2010 period and were aimed at the educational communities. The participants were: parents, teachers, students and local authorities such as mayors, councilors and District Directors of education, etc.
The seminars were offered at Vallegrande, Santa Cruz and San Rafael. Seminars were also held for high school students, called "Olympics of leadership". This was a very motivating and productive activity that sought to measure, in a comprehensive and dynamic way, the extent to which students took advantage of opportunities. The Olympics of leadership were held in Santa Cruz and Vallegrande. Curriculum seminars for teachers were also offered. The curriculum seminars were in the areas of mathematics and reading-writing. These seminars were offered in Riberalta, Rurrenabaque, Villamontes, and Santa Cruz.
Explanation on some causes that have prevented a significant percentage of students from finishing their studies:
a) Lack of documentation-A lot of teachers did not have their papers in order, especially the Title in National Provision.  The processing of the title has been very bureaucratic and has been delayed too much. Because of this requirement the teachers cannot finish their studies at the University.
b) Another reason for dropping out has been the distance of the educational units. Due to the distance there have been a lot of problems with communicating with the University and no doubt this has led to the discouragement of a percentage of participants.
Anyway, the percentage of desertion, reaching up to approximately 30%, is within the normal margins provided for training of this nature. I think that it would have been better to have training courses that are short, specific and targeted, not only to teachers but also to other members of the educational community.  And that is what we are thinking to do for the next five years with the extension. It is good to point out that it is perhaps better to decentralize trainings and offer them with institutions in each region.
The team mentioned the effort of teachers in innovative experiences being made, as is the case in the text drafting to support teaching mathematics, English, audiovisual materials, Christian education, teaching to deaf mutes, etc.  They are of great value.
One of the lessons learned is to generate processes of participatory planning to generate solid networks to strengthen processes of teaching - learning.
Likewise, it is necessary to point out that from the collected experiences; the importance of valuing the human factor as a fundamental aspect in developing processes in an effective and efficient way, is understandable, and will determine future decisions.

Next steps would be to undertake a process of socialization of the results and implementing the recommendations in the short, medium and long term plans.

Best Regards, 
PDI – BOL TEAM

Comments from Pym:
The evaluation has pointed out several important issues that need to be focused on for further improvement. The project staff and Pym’s partner have taken the recommendations seriously and are in process of improving internal routines, clarifying roles and responsibilities, collecting and systemizing data, decentralizing and improving internal monitoring, just to mention some of the actions taken so far.

It is worth mentioning, though, that during a dialogue with the evaluation team, Pym’s partner and the project staff shortly after the report was presented, it was clear that the positive findings had been much left out from the report, because as the evaluation team said; What is already achieved is not interesting, it is the areas for improvements that needs to be in focus. Consequently, according to Pym’s own impression (based upon regular communication, project visits and input from other stakeholders in Bolivia), combined with the external evaluation from 2009 and the oral information given by this evaluation team, the work of PDI-Bol deserves a more positive appraisal than what is found in this evaluation report. For instance, the cooperation with and ownership of relevant stakeholders (educational units (admin./leadership, teachers), parents, local officials etc) is significant. The flexibility and learning culture among project staff are other strengths. Thus, Pym is supportive to the comments given by the PDI Team above.