Project
Evaluating AERO’s evidence claims.
A systematic evaluation of the evidence cited in AERO’s (2023) report, How students learn best: An overview of the learning process and the most effective teaching practices.
do not match
do not match
are misused atleast once
AERO’s (2023) report contains 106 claims supported by 61 references. Where a claim cited more than one source, we evaluated it against each source separately (citation checks). A substantial share of the cited material does not support the statements attached to it.
Method
- Every sentence linked to a citation was treated as a distinct claim.
- Where one claim listed several sources, we evaluated it against each source separately. Each of those evaluations is a citation check.
- Five works were not listed in the reference list and could not be confirmed.
- Two NCCD citations were treated as references to the standards. Citations to ACARA and AITSL were kept.
- We read each cited source and looked for the closest available match to what AERO stated.
- When the relationship was unclear, we gave AERO the benefit of the doubt and recorded the claim as matching the source. The coding was therefore conservative.
- Each claim–source pair was categorised as Matches, Does not match, or Plausible.
How the percentages are calculated
- By claim: Percentage of claims that do not match at least one of the sources cited for them.
- By citation check: Percentage of individual claim–source evaluations that do not match.
- By reference: Percentage of unique references used at least once in a way that does not match the source.
- Popular-level books: Percentage of citation checks drawn from popular-level books - Willingham, (2021); Brown, (2014)
Because the coding rules favoured AERO in cases of doubt, these percentages should be read as conservative estimates.
Initial observations
- Even at this first level of analysis, of checking if what the citation says matches what AERO says, a substantial proportion of claims do not match what the cited sources say.
- AERO presents its work as drawing on research evidence, however a substantial proportion of the supporting citations come from popular and practitioner-ortiented books, summaries of research, explainers, and guidance reports. The majority of AERO's claims which match what the citation says come from these types of works
- This amount of mismatch does not appear to be limited to occasional referencing errors.
- Many cited sources are not original studies. They include reviews, explanatory articles, and popular books. Therefore, where AERO's claim matches what a cited source says, this does not mean AERO's claim follows from the underlying evidence from original studies.
- Early work on a second level of analysis, checking the primary studies behind key authorities, already indicates that some claims do not follow from the original evidence.
Sources prioritised for our deeper forthcoming analysis include frequently cited reviews, experimental work associated with cognitive load theory, and widely referenced popular books.
The purpose is not to single out either individual researchers or organisations . It is to apply a basic scientific standard: claims should be consistent with the evidence offered for them. Where they do not, that needs to be stated clearly, especially when such claims are used as a basis for systematically reshaping teacher education, school practice, and policy.
* Two references to the NCCD, n.d. were not included as they refer to the standards.