Report on the Mid Term Evaluation of the Education Program at SCI Cambodia

Background   
Save the Children in Cambodia is commissioning a mid-term evaluation of the education programme which has been implemented in six provinces; Kampong Cham, Kampong Chhnang, Preah Vihear, Koh Kong, Siem Reap and Pursat from 2011 to 2013, midway through the 2010-2015 strategic plan. Since the beginning of the Education Strategic plan in 2010, the education programme has undergone a large change process. Two different member programmes (Norway and Australia) have been brought under one management and one strategic vision. The education programme has also seen significant growth during that period both from traditional and non-traditional funding sources that have changed the nature of programming. New education projects have been implemented with adjusted implementation mechanisms and with increased focus on involvement of civil society organisations, communities and local authorities. 
The purpose of this evaluation is to gauge progress and challenges of all education projects within the education programme and to give recommendations for the further development of Save the Children’s education programme in Cambodia.
Purpose/objective (including evaluation questions)    
 Specific objectives:
Save the Children aims at evaluating the existing education programme, using the Theory of Change, in its entirety with the following objectives:
1. Assess the progress, outcomes and impact of the education programme against the set results and objectives of 2011-2015 long-term Education Plan, especially how effective and efficient the programme has been in its implementation
2. Assess the impact to the programme, positive and negative, of the change to one Save the Children
3. Identify the most significant factors that have constrained or contributed to the education programme’s achievements
4. Assess the current models of project implementation and delivery within the existing education programme in terms of efficiency, effectiveness and impact
5. Document evidence of good practice and innovative approaches

Methodology    
 Analysis of SCN SCI documents on the education program (Part 1); Survey research in target schools (Part 2). The survey methodologies include: interview with individual, focus group discussion, and consultative meeting with children, teacher, community, Save the Children Staff, and education officers at district and provincial education offices.

Key findings and Recommendations   
The report has two parts: Part 1 is a narrative based on an analysis of Save the Children documents and interviews with SCI staff and other stakeholders.  Part 2 is an analysis of the results of a field survey of target primary schools in the six provinces where SCI works.

Part 1
A close examination of Working Together to Make a Difference: Basic Education Development in Cambodia 2000-2010 reveals the development approach taken by SCN for many years before transition to SCI (which was finalized in November, 2011).  The SCN development approach shaped the roles, responsibilities, relationships, attitudes and practices in the Education Program.  Understanding this background helps clarify the challenges the Education Program will face as SCI adopts a new paradigm, represented by the The Education Global Initiative, Moving Ahead on Education, A focused strategy for achieving our goals 2012-2015.
The SCN approach grew out of the shift from relief assistance to development programming in the late 1990's and early 2000's.  This approach included the following elements:
• Partnerships were formed with the Provincial Education Office, including direct budget transfers to the partner and the placement of an SCN Provincial Officer at the PEO office to administer the details of the relationship.
• SCN in Cambodia became a donor instead of an implementer.  It delegated responsibility for program activities to the partner; but the government partner could not be held accountable by the NGO.
• To avoid creating dependency and to assure partner ownership, SCN rejected the "project mindset" and instead promoted activities along general avenues of concern: Access, Quality, Improvement of Education, and Strengthening Systems.
• SCN approached capacity building through the use of "natural teachers" within the ranks of the government partner.  This was culturally reasonable, as appearing to suggest that a high ranking government official might need training or instruction would be considered an offence to his status.  But the result was persisting low capacity in the partner.
• SCN successfully leveraged its close partnership with the Ministry of Education, Youth and Sport to introduce Child Friendly School frameworks, until they became government policy.
• SCN favoured concise reporting that followed the Log Frame, and stated numerical results against targets and that included lists of the main activities.  The Education Program team did not get experience in writing detailed reports on program activities to document the successful program implementation activities that they had devised.  As a result, many innovations, lessons learned and best practices were never recorded. 

Recommendations:
• This rich program implementation experience remains in the minds of the senior team members, some of whom have been with Save the Children for over twenty years.  Means should be found to transmit this knowledge systematically to the younger generation of SCI Education Program team members.
• Building the capacity of sub-national administrators of education is important, but the SCN approach should be reconsidered.  The younger generation of Ministry staff may be less threatened by the prospect of professional training to improve their job performance.  They have grown up in a more stable and peaceful environment than their elders, who experienced the terror of Pol Pot, the occupation by Vietnam and civil war conditions during their careers.
• Expert training that is culturally sensitive is needed on all aspects of the management and administration of a modern national education system.
• Capacity building of teachers is a task for the Teacher Training Colleges which have now been established in all provinces.  This will assure that each new cohort of teachers will be reached pre-service with training in modern pedagogical skills, rather than an uneven coverage of teachers in-service in the target schools.  This approach should replace the ineffective impact of Provincial and District Education staff, who are themselves struggling to understand and implement the new concepts.
• The long-established relationship with Provincial Education Offices should be preserved, but modified in light of a more careful assessment of the specific needs of each province.

The transition to one Save the Children and the publication of the new vision for a global initiative based on the Theory of Change will bring new challenges and opportunities to the Education Program.  An analysis of the legacy of the past and the implications of transition to one SCI suggests that there are three main areas of change that the Education Program will face:
1.  The first change is a return to implementation at the school level, especially in connection with the new signature program "I'm Learning," and a shift from an emphasis on access to greater concern for quality with the adoption of the Quality Learning Environment (QLE) approach.
• This will call for relationship building at individual schools, which will require a new level of detailed record keeping of interventions.
• This approach will create new staffing and logistical challenges to support local level implementation.
• The QLE tools and the global standards will introduce a new level of scrutiny on the gathering and compiling of data.  Global standards will require heightened accuracy and accountability in monitoring, evaluation and reporting on implementation efforts.
• The emphasis in the "I'm Learning" approach is on teacher-student relations and the characteristics of good teachers.  Many of the principles in "I'm Learning" resonate with traditional Buddhist principles (loving kindness, sympathetic joy and equanimity).  The new signature project being piloted in Cambodia gives the Education Program the opportunity to make innovative contributions to the experiment.

2.  The second change for the Education Program is a new emphasis on evidence and reporting.
• Detailed evidence on all interventions, whether successful or not, should be gathered and reported.  Reporting in depth should encourage the documentation of innovations, lessons learned and best practices. Writing reports of this kind will require, and build, critical thinking about program implementation.
• SCI is a knowledge-based organization that values documenting, reporting, filing and sharing findings to facilitate a global dialogue among education specialists and to preserve institutional memory of what worked, and why.
• Persuasive, detailed reports on Education Program activities will support SCI proposals to donors in an increasingly competitive environment for NGOs.  It is no longer likely that Save the Children country offices will depend on a single donor for decades.

3.  The third change for the Education Program is to shift away from a Log Frame mentality of "actions taken and results achieved" to Theory of Change thinking that asks, "why do you think that this action will result in that change?"
• At the Global level, the Theory of Change states the highest principles of the organization.  In SCI, partnership, innovation, voice of children and achieving results at scale all depend on using an evidence-based approach to support advocacy for better practices, documentation of program effectiveness and breakthroughs.
• At the Country level, the Theory of Change inspires a concern to make assumptions explicit.  This process enhances skill in critical reflection, which is needed at every phase of program implementation, planning, adapting to circumstances and writing clear and thoughtful reports.
• At the Program level, an exercise in Theory of Change thinking can provide a means by which the Education Program can contribute to the pilot "I'm Learning project."  The pilot asks countries to develop appropriate activities to advance Life-Skills, one of which is "critical thinking" in primary school pupils.

Recommendation:
• The Theory of Change exercise recommended for the Education Program team is to ask, "what activities or interventions in a Cambodian classroom would achieve the result of increased critical thinking; and why?" Another aspect of this exercise would be to ask, "what preconditions are needed in order for the result increased critical thinking to be achieved in the Cambodia context; and why?"

Part 2
The survey research was designed to assess the progress, outcomes and impact of the Education Program activities in target schools.  The activities were listed in the Thematic Long term plan – Education, 2011-2015 Early Childhood Care and Development and Basic Education, and were based in the SCN development approach prior to transition to one Save the Children.
• The analysis of the survey data is presented in detailed charts and cross tabulation tables.  This material is grouped by type of intervention, School Infrastructure, Access and Quality.  A dashboard provides a summary, by province, of high, medium and low impact results for each activity.
• The survey study found that at the country level, the Education Program had been very effective in implementing the activities of the Long term Plan Log Frame.  This is confirmed by comparing two categories of target schools—high and low level of implementation schools; the former showing significantly higher scores than the latter on most Log Frame activities.
• The two categories of school actually reveal an uneven effectiveness of the Education Program impact.  Interventions have been more successful at some focus, or core schools.  But at schools which received less direct attention, and which depended more on the cascade effect facilitated by the implementing partner, the District Education Office and Provincial Education Office, the impact was reduced.
• At the province level, the analysis of survey results revealed deep disparities in program impact.  These differences are obscured at the "average" or country level.  Drilling down into the data shows a complex pattern of significant differences between strong and weak performers on most activities.
• The unevenness in impact at the province level reflects inefficiencies.  The Education Program might be much more efficient if it took these differences into consideration as it develops the annual agreement and work plan with the PEO based on the needs and progress in each province.

Recommendations:
• The survey study illustrates an evidence-based approach that can provide information to Education Program managers enabling them to adjust support strategically and to guide partners toward achieving impacts in each province that contribute to a more balanced impact across the target provinces.
• A new, differentiated approach to the Provincial Education Office partners would tailor support to the needs, challenges and performance in each province.  At the same time, the autonomy and sovereignty of the local government authority must be respected, maintaining the close and long-enduring relationship.  The Education Program team has the depth of local experience and political wisdom to manage these relationships with careful balance.

Recommendations   
 See above

Follow up (with reference to Action Plan) 
 The evaluation report will widely share with the Save the Children staff and partner, especially during the project and education programme review and planning.