Dr David Clements, a research fellow at Victoria University, and Dr Nataša Ciabatti, an education lecturer at Federation University, told EducationHQ they launched ACPUE because they wanted to examine the research claims that are currently shaping policy and practice in Australian schools.

“The concern is that the content and ways of teaching that are now being imposed system-wide, including in initial teacher education, are being justified in the name of evidence and science, rather than necessarily aligning with the evidence,” they said.

“…We are committed to a basic standard of inquiry that if a claim is offered as evidence-based, it should follow from the evidence attached to it, and it should remain open to revision.”

In a new analysis of AERO's 2023 report ‘How students learn best’, ACPUE check the claims made against supporting references.

The Centre found a ‘substantial share’ of the cited material did not support the statements attached to it.

“It was a logical starting point for several reasons,” Clements and Ciabatti say of the AERO report.

“The title itself is a large claim: how students learn best. If that is the claim, there had better be research behind it. AERO is a national education research organisation, not only a practitioner advice service….

“We work in Australian universities, and in our conversations with people in education systems the report comes up as a basis for other work.

“System-wide change is being implemented with reference to this body of work. For all of those reasons, it was the right place to begin for our method. Our published analysis is of course not a sample of all of AERO's output.”

The method was fairly simple, they add.

“We treated every sentence tied to a citation as a claim. Where a claim listed several sources, we checked each source separately. We read the cited works and looked for the closest available match to what AERO said.

“On our first-level analysis, 47 per cent of claims do not match at least one of the sources cited for them; 42 per cent of citation checks do not match; and 67 per cent of unique references are used at least once in a way that does not match the source.

“About 15 per cent of the citation checks come from two popular-level books.”

The pair say the important finding is “not a fight over a percentage point but the extent and the pattern”.

“This does not look like the odd referencing slip. A substantial share of the cited material does not support the statements attached to it.

“Furthermore, many of the sources are not original studies. They are reviews, explainers, guidance reports and popular or practitioner books….”

Further analysis now underway is checking the primary studies behind key claims made by AERO, and already it’s been found that some do not follow from the original evidence either, they add.

But Dr Greg Ashman, deputy principal at Ballarat Clarendon College in Victoria, has called ACPUE’s first published analysis ‘a hatchet job’ and said it contains genuine issues.

He maintains the analysis is really an attempted take down of AERO.

“…Many claims have multiple references and each individual reference is analysed independently. This leads to an odd phenomenon. If AERO writes, say, ‘(Smith 2012, Jones 2015, Brown 2021)’ at the end of a passage of text, and if ACPUE judges that each of Smith and Brown ‘matches’ the claim, it will still list Jones as ‘does not match’ unless it encompasses all parts of the claim.

“This is uncharitable because the AERO authors have supported the claim with the other two papers and Jones may only be relevant to a part of it,” he wrote in a recent Substack post.

Ashman noted that AERO has many critics in the faculties and bureaucracies of Australian education.

 “I suspect these are the people who prefer problems to solutions and see AERO, with its clear, practical guidance, as a direct threat.

“In a world where it’s all too hard, AERO proves you can just do things. How is a savvy administrator expected to hold on to that lucrative sinecure if problems are simply solved?” he posed.

Last year a group of academics and teacher educators took aim at AERO in articles published on the AARE blog, arguing the government-funded organisation was fundamentally harming teachers’ autonomy and being used by governments to push a ‘neoliberal’ agenda in classrooms.

AERO's CEO Dr Jenny Donovan has called out those in academia who remain invested in discredited instructional approaches and that openly criticise practices that align with what we know works best.

“It’s kind of a mystifying position to me. I don’t really understand it,” the former high school teacher told EducationHQ earlier this year. 

“I think it mostly comes from people who feel very comfortable with the way things are, and what they’re currently doing, and they don’t like the suggestion that things could be better, things could be improved, or that people might need to make changes.”

Since its establishment in 2021, AERO has advocated for a system-wide shift to explicit teaching “because that’s what the evidence is most strongly behind”, Donovan added. 

“The evidence-base for explicit and systematic approaches is overwhelming and robust. 

“So right from the very beginning, we’ve been very clear that this is what the evidence points to – and we’ve made it our business to unpack what that means for people in education, whether they are classroom teachers or system leaders or ministers…”

ACPUE list a number of upcoming projects, including analysis of the evidence behind the TEEP report, an examination of AERO materials that operationalise the Victorian Teaching and Learning Model 2.0, and an evaluation of foundational studies 'frequently cited on cognitive load and worked examples'.

Ashman says these projects clearly show it is not an impartial actor.

“…the Centre has issues with anything broadly related to what’s become known as the ‘science of learning’. There’s nothing intrinsically wrong with that, but presenting the exercise as a neutral-sounding attempt at ‘public understanding’ seems a little misleading to me,” he argued.

Clements and Ciabatti said that readers could decide for themselves whether their analysis of AERO's claims were correct or not.

“AERO was first because the report is usable as a public test, not because the Centre exists to defeat AERO.

“The same method will be applied to other influential texts, including documents that are not part of the current science of learning approach. Again, those evaluations can be judged when they appear,” they added.

The academics maintain they would “rather be useful than merely oppositional as plenty of critique already exists”.

“What we are trying to build is a public resource of demonstrations people can read, check and use to see for themselves,” they noted.

“What's more, our analyses can be revised in our response to public engagement and we will revise any of our analysis when a better reading of a source is shown.

“This comes back to our understanding of education as public – being open to revision is part of what it means to be educational.”

The Centre is not a part of or affiliated with any university. While it is currently unclear who else is involved in its work, the website notes an advisory group is being formalised with an update to come.