Impact evaluation of a literacy intervention on teacher training and teacher coaching

Abstract

 Presenter (s) Laura V. Sánchez-Vincitore - Universidad Iberoamericana (UNIBE); Aída Mencía-Ripley - Universidad Iberoamericana (UNIBE); Clédenin Veras - Universidad Iberoamericana (UNIBE); Sonia Molina - Universidad Iberoamericana (UNIBE); Maritza Cabrera - Universidad Iberoamericana (UNIBE); Carlos Ruiz-Matuk - Universidad Iberoamericana (UNIBE)
Title Impact evaluation of a literacy intervention on teacher training and teacher coaching

This paper presents the results of the first impact study of the project [anonymous]. This project is an initiative of [anonymous] that attempts to improve literacy levels of 200,000 public school students in the Dominican Republic in five years.

To determine the intervention effect, we conducted an experimental study. The study had measurements before and after the intervention, and with an experimental and a control group. As of now, the project has collected data at baseline and midline. Given the characteristics of projects that involve the cooperation of MoE, [anonymous], and academia, the intervention during the first two years was based on teacher training and teacher coaching. However, the original intervention included the incorporation of leveled reading practice materials that reached the schools after the midline evaluation. Therefore, this study aims to determine to what extent a teacher training and coaching intervention impacts reading skills.

The theoretical framework that drove the intervention is based on the Simple View Theory of Reading (Hoover & Gough, 1990) that proposes that reading comprehension depends on oral comprehension and word recognition. As such, we operationalized this model by the evaluation of sub-skills associated with each model component. In addition, we are considering the Theory of Reading Automaticity (LaBerge & Samuels, 1974; Samuels, 2004) that proposes a cognitive load model. In this model, reading ability is considered a complex skill that overloads working memory. In order to read for comprehension, reading subskills such as decoding must be automatized. In other words, these must be performed without conscious effort. To create the reading intervention, we revised the literature on stages of reading acquisition by multiple authors (Ehri, 1987, 2005; Ferreiro & Teberosky, 1979; Frith, 1985; Pressley, 2002; Walker, 2012). Also, we revised the current literature on the neurobiology of literacy acquisition (Dehaene et al., 2010).

The initial phase of the intervention included the work with teachers. Teachers were trained on evidence-based practice twice a year for 8 hours each training. This training consisted of helping teachers understand the Dominican curriculum, emphasizing communicative competence, and the use of text diversity in literacy instruction. Also, reading sub-skills and classroom activities to improve them.

Besides, teachers received seven mentoring sessions a year. The mentoring activity consisted of a visit by the project personnel to observe teachers while they were in the class with their students. They were monitored by a mentor who used a standardized monitoring instrument. The mentor and the teacher would have a reflexive dialog to discuss the evaluated class.

Data collection occurred in two stages: baseline in 2015 and midline in 2017. Participants were students from 400 schools (200 control schools and 200 experimental schools). We obtained a probabilistic sample of 2,400 students (1,200 control students and 1,200 experimental students) on each evaluation stage. Six students were randomly selected from each school to participate in the evaluation.

We evaluated the following reading abilities: oral comprehension, phonological awareness, letter knowledge, words per minute, pseudo-words per minute, fluency, and reading comprehension. In addition, we evaluated working memory, a general cognitive ability necessary for any learning process.

The results showed that students from the experimental group outperformed the control group on working memory for pseudo-words, oral comprehension, phonological awareness, and words per minute. However, there were no statistically significant differences between the groups on reading fluency and reading comprehension.

These results suggest that the intervention on teacher training and teacher coaching improved student literacy skills on the very basic levels, specifically in phonological processing. This is evidenced by changes in pseudo-word processing and phonological awareness. In addition, this intervention was able to improve students’ oral comprehension, which is a crucial component of the Simple View Theory of Reading. Students also improved their word recognition abilities. Nevertheless, the intervention has not reached the point of improving more complex reading skills such as fluency and reading comprehension. In other words, the intervention was successful at improving the components of literacy but has not been successful at integrating these individual components into one single skill of reading comprehension. It is important to note that reading automaticity (manifested as reading fluently and with basic comprehension), only occurs after massive amounts of practice. Since the leveled practice materials arrived in the schools after the midline study, we expected to find improvements in reading skills at the basic levels.
Teacher training and teacher coaching showed to be an effective strategy to teach the reading code and to establish solid reading pre-requisites, including language comprehension and word recognition, but it is insufficient for fostering reading automaticity and therefore reading comprehension. In order to accomplish this, students must be given high-quality materials to practice their reading skills, in addition to explicit and implicit instruction by teachers.

– Laura V. Sánchez-Vincitore is a psychologist. She is the director of the Neurocognition and Psychophysiology Lab at UNIBE, and Project USAID Read’s research advisor.
– Aída Mencía-Ripley is a psychologist and currently the Dean of Research at UNIBE.
– Clédenin Veras is an educator. She is Project USAD Read’s chief of party and the director of the School of Education at UNIBE.
– Sonia Molina is an educator. She is Project USAID Read’s teacher training and teacher coaching specialist.
– Maritza Cabrera is an educator. She is Project USAID Read’s literacy specialist.
– Carlos Ruiz-Matuk is a psychologist. He is Project USAID Read’s methodologist.

1 Response

  1. Carol Benson

    Thank you for this poster!
    It actually deals directly with one of the most urgent critiques of EGRA and associated Simple View pedagogies: That they are not suited to contexts such as these, that they have a negative backwash effect on teaching (focusing too much on phonemic awareness and “fast” reading) and not working at all with WRITING, or other communicative and productive aspects of reading.
    So even with Spanish, a fairly regular language with “transparent” orthography, phonics is not enough. Seems obvious to many of us, but this feedback needs to go back to USAID as a direct critique to its highly limited approach as driven by EGRA.
    Some of us scholars are gearing up for published academic articles demonstrating the harm that EGR interventions are doing in multilingual countries. I hope you will also publish your research. Thank you!

Leave a Reply