High Fidelity: The importance of efficiency and efficacy for early grade reading programs

Abstract

 Presenter (s) Simon King, RTI International; Gilbert Jolly, Jolly Learning; Jake Thomsen, Chemonics International

Expansion of school enrollment has increased in over the past 30 years in many developing countries; however, the quality of education has stagnated (UNESCO, 2006).

Recognizing the importance of early grade reading to lay the foundations of improved pupil retention and achievement, many governments and development partners have invested in early grade reading programs in order to help children to “read to learn”. A great challenge of these, often at-scale programs is fidelity of implementation – that participants are implementing the reading program components as intended.
Measuring fidelity can happen at two stages; during implementation and monitoring, and/or part of program evaluation.

Fidelity of Implementation is a ‘relatively’ new phenomena in research, not developed as an idea until the 1970s in public health (O’Donnell, 2008). Prior to its development, researchers generally considered fidelity to be high and participants to be accepting implementors of an innovation (Rogers, 2003). Published research on fidelity in education suggests that FOI is multi-dimensional and can be separated into two general criteria (Mowbray et al, 2003);
– fidelity to structure (duration of instruction); that the teachers are teaching according to the program components; for example, using the lesson plans and recommended activity duration.
– fidelity to process (quality of instruction), and that the teachers are implementing the program using the appropriate techniques; for example, giving students adequate time to individually practice reading skills and receive feedback.

The Education Sciences Reform Act of 2002 (U.S. Department of Education, 2002) identified one component of scientifically valid education evaluation as one that “ . . . to the extent possible, examines the relationship between program implementation and program impacts”. It requires researchers to report not only on the impact of education programs; but how well the participants (teachers, headteachers, supervisors, etc.) implemented the program. Carrol et al. (2007) noted that without a FOI measure, evaluations cannot inform why a program has failed to produce impact or how to further enhance a program even if program impact is established. Key to this assessment is the separation of efficacy and efficiency of the program; in other words, how well the program’s intervention components are working as intended and how well they are being implemented. FOI is usually concerned with the latter.

Therefore, while reporting FOI during an evaluation is critical for considerations such as scaling-up, FOI is also measured during routine monitoring of education programs. By monitoring how well participants implement the program down to the school level, monitoring fidelity considers the individual human aspect; rather than generalizing fidelity through evaluation, program monitoring of fidelity provides an individualized solution to support all participants and stakeholders implementing the program. However, at-scale this support can be messy and complex; each teacher exists in their own environment and has their own unique sets of challenges. Thus, an adaptive management approach is taken; adding flexibility to support participants with their individual needs.

The four presenters on this panel explore different aspects of fidelity of implementation; ranging from evidence to support how we need to better define FOI in a low-income country context, how FOI measures were used in country-specific contexts and the use of cutting-edge technology to better monitor programs.

The first panelist discusses the challenge that the definition of FOI for low-income country education programs has been only narrowly adapted from domestic US education and lacks contextualization. For example, domestically – time-on-task is usually measured through classroom teaching observation; however, while teacher absenteeism from the classroom is around 5% domestically (National Council on Teacher Quality) it is frequently over 50% in sub-Saharan Africa classrooms (World Bank, 2018); demonstrating that our multi-dimensional definition and measurement of FOI needs to expand beyond just measurement process and structure to the program in the classroom. The panelist will further explain why we traditionally have not expanded our definition and presents a process in which we better contextualize FOI for low-income countries and enhance FOI on reading programs.

Our second panelist explores FOI outcomes for program efficacy and efficiency from the USAID and DFID jointly funded early grade reading program in the Democratic Republic of Congo (DRC). The presentation will emphasize the “real world” context of the implementation of an early grade reading program where two Ebola outbreaks, militias, and government regulations have created the need for implementors to adapt their approach without compromise in using FOI measures to understand what works in the reading program and where opportunities exist for program enhancements.

The third panelists will present learning outcomes and discuss how measures of fidelity were utilized from a USAID-funded early grade reading program in Senegal. The program’s core literacy approach requires a high fidelity of implementation and so the program is working with the ministry to measure to investigate FOI issues to adapt the program and link lessons learned to policy recommendations for sustainability and scale-up. The panelists will discuss midterm findings as well as its coaching model which applied blended ICT support.

Our fourth panelist discusses FOI by presenting how a monitoring system is being used to enhance the “Jolly Futures” project – a literacy project that has reached tens of thousands of public schools across sub-Saharan Africa and South Asia. The monitoring system, designed by incorporating feedback from stakeholders and users incorporates lessons observations and classroom assessments into a smartphone app and dashboard which can be then used by teachers, headteachers, and coaches to monitor the progress of pupils’ learning outcomes and classroom instruction. Coaches are empowered to view all the data relevant to their work, including prior visits to teachers to better monitor progress. The panelist will discuss rollout, adaption, and usage of the system to better inform stakeholders of progress and improvement of implementation fidelity.

The final panelist presents on how to design monitoring systems that are sustained by ministry personnel beyond the life of the donor-funded early grade reading program. Aligned to “Journey to Self-Reliance” – design considerations include zero-cost tools with simple back-end usability such that the ministry can independently maintain and enhance their monitoring tools.

Leave a Reply