The longitudinal meta-analysis canvassed the findings from a total of 19 studies on Reading Recovery’s (RR) efficacy, concluding short-term positive outcomes from the program were ‘negligible’.
Researchers also found evidence to show students who received RR instruction actually performed worse over time compared to those who did not.
A question of statistical significance
Lead researcher, teacher and author Nathaniel Hansford told EducationHQ that although most of the numerous studies on RR have reported positive outcomes, these results have not always been statistically significant.
“When we analysed the average effect across studies, we found that the mean effect size was negative,” Hansford explains.
“This suggests that students who participate in Reading Recovery may fare worse in the long run than those who do not.
“This is particularly surprising given that the program provides intensive one-on-one reading instruction.”
Yet there are some caveats to flag here, the researcher says.
Namely, some longitudinal studies have reported positive effects, and many have used non-equivalent group designs, which compared struggling students receiving RR to non-struggling students who received none.
Hansford says these studies may have 'deflated' RR's recorded impact.
“One key takeaway from our meta-analysis is that there was little evidence of researcher bias in Reading Recovery studies.
“This is unusual in education research, where publication bias often inflates reported effect sizes...” he notes.
RR under the microscope
Founded by Marie Clay in the 1970s, RR is a ‘constructivist inspired’, Tier 3 reading intervention that commonly offers 30 minutes of daily one-on-one reading instruction to struggling readers, run over a course of 12-20 weeks.
The program has faced intensifying criticism and scrutiny in recent years, with dyslexia advocacy groups, researchers and other literacy experts arguing it fails to teach phonics systematically and relies on discredited instructional approaches, such as three-cueing, and less effective resources, such as levelled readers.
James Chapman, Professor Emeritus of Educational Psychology at Massey University, has previously told EducationHQ that RR’s claimed success as an early literacy intervention is “somewhat fraudulent” and has urged school principals to ditch the program without hesitation.
According to Chapman, RR’s claimed efficacy is “the biggest myth in New Zealand education”.
“[That’s] actually a polite statement…” he warned.
“It's leading to false hopes for many, many children and for their parents. Reading Recovery is not being upfront about the limitations of the program, and about the extensive research conducted independently by researchers across a range of countries showing that the program does not live up to its claims.
“To that extent, the claims appear to be somewhat fraudulent. I just don't see it any other way.”
A note on fluency
The meta-analysis found that while RR may have some impact on lifting children’s reading levels, it was less effective in building fluency skills – an essential component of skilled reading.
This was one of the most striking findings, Hansford says.
“This distinction (between reading levels and fluency) is important because many reading level assessments, like the BAS test, are notoriously unreliable,” Hansford begins.
“For example, one study found that the BAS was no more accurate than flipping a coin.
“In contrast, fluency assessments such as Acadience and DIBELS are norm-referenced and provide a more objective measure of reading ability, factoring in both accuracy and speed...”
Since fluency is a strong predictor of long-term reading success, RR's poor performance on fluency measures is 'concerning', the researcher warns.
The study also found other types of assessments, such as those measuring students' comprehension, spelling, and decoding, showed mixed results, "with some effect sizes being close to zero or even negative, further complicating the interpretation of RR’s effectiveness”.
However, researchers noted the program does provide some benefits.
“When the authors discussed RR with RR teachers, they were enthusiastic in their support of RR and cited extensive training and collective self-efficacy as being positive elements to the program,” the paper reports.
Researchers concluded that while it "may be likely that RR is not best practice as it currently stands, there is strong evidence that it can be improved to better support students”.
Call for more explicit instruction on phonics
Yet red flags were drawn against RR’s overall approach to reading intervention, which the authors argue most closely aligns with Whole Language and not a systematic phonics approach – what we know works best to lift reading outcomes.
“RR should adopt a scope and sequence that outlines the specific letter sound correlations that should be taught, the order they should be taught, and the pace at which they should be taught,” the paper recommends.
“Furthermore, RR should provide more explicit instruction on phonemic awareness and phonics.”
A complete overahaul of the program is not needed, Hansford says, with these changes being "relatively simple and easy to implement”.
Yet the researcher recognises that these recomendations might prove 'philosophically challenging' for some RR advocates.
“Reading Recovery already does many things well. It provides students with intensive early intervention and supports fluency, vocabulary, and comprehension.
“Research... has consistently shown that early intervention in Grade 1 is the most effective time to support struggling readers,” he adds.
The 'AND + AND' approach
In 2023, New Zealand’s RR program underwent a refresh, but questions linger as to whether the changes went far enough.
Rather than align its approach entirely with structured literacy, curriculum designers at Reading Recovery said both methods could be used in conjunction.
In a blog post on the Tui Tuia Learning Circle website, it explains:
“Our approach is to reject a binary or oppositional approach, and so, we are implementing an AND + AND approach,” wrote RR Reading trainer and Associate Professor at University of Auckland, Dr Rebecca Jesson at the time.
“In Aotearoa, we are using the expertise of phonics AND the expertise of Reading Recovery.”
But Chapman deemed the new AND + AND approach as an “awful mix”.
“It’s kind of like mixing new wine with old wine and putting that mixture in used wine bottles.
“The worst parts of Reading Recovery, which have demonstrated not to benefit kids who go into the program who are struggling the most with learning to read, still continue.”
A look at predictive texts and three-cueing
On the issue of RR using non-decodable texts, the meta-analysis threw up some surprising results.
Namely, that evidence showing that decodable texts are more effective than predictive texts is not clear-cut.
It highlights one study from last year that found no strong evidence to suggest that only decodable texts should be used with beginner readers.
“Indeed, they found the highest results when both decodable and non-decodable texts were used,” researchers note.
According to Jesson, decodable texts offer opportunity to practice taught skills, "but not enough opportunity to meet common spelling patterns or the language structures and vocabulary of English. So, both decodable texts and guided reading are possibilities".
When it comes to assessing the efficacy of the three-cueing method, the study warned that no peer-reviewed meta-analyses currently exist, making it tricky to drawn definitive conclusions.
“Although it may be likely that RR is not best practice as it currently stands, there is strong evidence that it can be improved to better support students,” the researchers reiterate.
In the Australian context, Reading Recovery Australia promote the program's effectiveness online and share links to various studies to demonstrate its positive impact.
“Over decades of research and evaluation results, Reading Recovery is the world’s most widely studied early reading intervention program,” the site reads.
“The effects of Reading Recovery have been examined in high-quality experimental and numerous qualitative studies.
“Reading Recovery appears to be effective in supporting the literacy development and educational achievement of children taking part ... and is an 'effective intervention improving short-term reading outcomes among poorest performing readers, which is the intention of the intervention'.”
