A Program Evaluation of an International Collaboration to Better Train, Support, and Prepare Teachers in Saudi Arabia

Abstract

 Presenter (s) Byoung-gyu Gong, Arizona State University; Clarin Collins, Arizona State University; Audrey Amrein-Beardsley, Arizona State University; Alejandra Enriquez Gates, Arizona State University; Ruhi Khan, Arizona State University

The Kingdom of Saudi Arabia (KSA) is envisioning a future less dependent on oil, with a diversified industry harvested through quality education and a knowledge-based economy. The KSA’s “Building Leadership for Change Through School Immersion” program was accordingly designed to send at least 25,000 KSA teachers, counselors, school leaders, and principals abroad to participate in one-year university-guided immersion programs in K-12 schools, until 2030, with the goal of reforming the education system in the KSA.

For this study, scholars evaluated the objectives and outcomes of the KSA participants as per the goals of the Ministry of Education (MoE) of KSA. More specifically, evaluators sought to measure to what extent the program had an impact on the KSA teacher participants, as per the purposes and priorities of the KSA and its MoE. Hence, evaluators assessed program impacts in terms of changes in: (1) teacher efficacy, (2), knowledge and experience related to program specific targets; (3) comparative educational understandings via the collection of data through pre- and post-surveys; and (4) English proficiency via pre- and post- English proficiency test scores. Evaluators assessed each of these areas of interest to determine whether the expected outcomes were achieved via this program’s interventions. In addition, evaluators interviewed a handful of participants, and examined (5) policy understandings of the teachers; (6) memorable events in the program; and (7) KSA teachers’ plans for applying learnings back in KSA.

Evaluators used three data collection methods: (1) pre- and post-surveys, (2) pre- and post- English tests, and (3) in-depth interviews. For the pre- and post-surveys and English tests, and with intention to capture the effect of the program’s interventions, evaluators adopted a one-group, pretest-posttest design; a quasi-experimental design method (i.e., given there were no comparison groups to whom evaluators could compare KSA participant impacts). For the pre- and post-survey, evaluators selected all 46 KSA teacher participants as their sample, which represented the entire population of those involved in this particular immersion program. To capture the effect of program interventions across multiple dimensions, evaluators constructed survey instruments with three different sections which related to each evaluation question. These sections pertained to participants’ changes in (1) teacher efficacy, (2) program specific outcomes, and (3) comparative understandings of education. Evaluators calculated statistically significant changes over time, and effect sizes using Cohen’s d (Cohen, 1988, 1992) to quantify the practical difference, or pragmatic changes between pre- and post-survey results.

For the English tests, evaluators used another one-group pretest-posttest design, akin to the quasi-experimental design used for the survey components. The KSA teacher participants took their first English test, the Michigan English Placement Test (formerly known as Cambridge Michigan English Placement Test, or CaMLA; Walter & Hentschel, 2013) towards the start of the program and completed their posttests following four months of intensive English training provided via the larger university. The tests assessed listening (including grammar, vocabulary, reading comprehension) and speaking (including grammar, syntax, usage and picture identification). Again, evaluators calculated statistically significant changes over time, and effect sizes using Cohen’s d (Cohen, 1988, 1992).

Evaluators also conducted interviews with a handful of self-selected KSA participants in order to examine their more in-depth program experiences, perceptions, and opinions. Evaluators more specifically focused on three different topics, as also aligned with their evaluation research questions related to (5) teachers’ policy understanding, (6) memorable events in the program, and (7) plans for applying learnings back in KSA.

Overall, evaluators found that participating in the immersion program increased KSA teacher participants’ (1) senses of teacher efficacy, (2) program specific outcomes, and (3) comparative understandings of education as per the survey results. More specifically, KSA teachers showed significant and moderate effect sizes in their teacher efficacy (d=0.47 and d=0.62 for English-language teachers and non-English language teachers respectively), while program specific outcomes (e.g., technical competencies) showed smaller effects (d=0.10 and d=0.20 for English-language teachers and non-English language teachers respectively). Participants showed most progress in their compartative understandings of the US education system, yielding large effect sizes (d=1.07 and d=0.64 for English-language teachers and non-English language teachers respectively). However, participants did not yield any changes in their perception of mobilizing knowledge and transferring skills learned in the US. These results showed no statistical or practical significance.

Evaluators also observed strong evidence that English language instruction improved teacher participants’ English language proficiency, again, as measured and assessed through listening and reading (including grammar, vocabulary, reading comprehension) and speaking (including grammar, syntax, usage and picture identification), and writing. Evaluators found that each of the elements assessed via the aforementioned English test had practical significance with decent-to-good effect sizes (e.g., d=0.27- d=0.48). A third portion of the English test designed by the hosting institution had students write responses to prompts and questions; “answering questions” yielded a moderate effect size (d=0.65).

Lastly, evaluators found that teacher participants understood the KSA government’s reform as an effort to transfer educational features from nations with advanced economies; although, KSA teachers’ sources of information regarding the reform were mostly informal and their interpretations of said reform measure varied. Second, KSA teacher participants commonly selected the immersion experience as the most inspiring and motivating source of learning throughout the whole program. Third, when asked about transferring knowledge back to the KSA, teacher participants mostly responded that they would focus on building better and more individualized relationships with their students to improve the educational experience for all of their students back in KSA.

These findings are significant given their contributions to our collective understanding about the impacts of teacher study abroad and immersion professional development programs, especially amid a time of increasing internationalization and mobilization. This evaluation research is also significant in that results might provide direct insights into the types of formative feedback needed by others who are becoming more internationally focused. Research or evaluation studies on teacher study abroad professional development programs should ultimately help all involved improve similar programs and services.