Hatties Defenses

Hattie (2010). On Being a Critic and Conscience of Society - response to Snook et al. (2009).

Hattie & Topphol (2012). Dialogue with Topphol & his stats students.

Hattie (2015). Corrections in VL2 - response to Topphol.

Hattie (2017). Educators are not uncritical believers of a cult figure - response to Eacott.

Hattie & Lovell (2018). Podcast Interview with Lovell response to Simpson.

Hattie Interview Pedagogy Non Grata (2019) - Podcast response to Wiliam. 

Hattie (2019). Defends again Romer's critique - response to Romer.

Hattie & Hamilton (2020). Real Gold vs, Fool's Gold - general defense.

Hattie & Hamilton (2020b). As Good As Gold - general defense.

Hattie & Larsen (2020). The Purposes of Education - dialogue with Larsen.

Kraft & Hattie (2021). dialogue with Kraft.

Summary of Issues in Visible Learning

Numerous peer review critiques detail a list of specific examples of issues, e.g.,

Misrepresentation or the use of studies not measuring the influence in question, e.g., Self Report Grades, Feedback & Behavior, etc.

Use of studies on non-school people, .e.g., doctors, tradesmen, military personnel, university students, etc., e.g., Feedback, Problem Based Learning & Cohesion, etc.

Use of specific populations of students such as those with learning disabilities, e.g., Reducing Disruptive Behaviour & Self Report Grades.

Standardised tests can get 1/5th the ES compared with a specific curriculum based test, for the same intervention. So Hattie's comparing ES from different tests is totally misleading. 

Many calculation & other errors, e.g., reporting negative ES when they are positive.

Use of studies not measuring achievement but something else, e.g., behavior, engagement, IQ, etc., e.g., Reducing Disruptive Behavior.

The equal weight of meta-analyses involving from 4 to 4000 studies.

Averaging many studies to get 1-Effect Size loses the meaning and detail of the studies.

Major issues of range restriction and control groups which have been shown to significantly change effect size calculations. 

The problem of the age of the students and the time over which studies ran.

Summary of Hattie's Defenses

Strategy 1 - Initially, Hattie consistently claimed that no mistakes have been found in his work, e.g., 

"What I find fascinating is that since I first published this back in the 1990s, no one has come up with a better explanation for the data... 

I am updating the meta-analysis all the time; I am up to 1400 now. I do that because I want to be the first to discover the error, the mistake." (Knudsen, 2017, p. 7).

Strategy 2 - Don't address the specific issue, e.g., defending against the many critiques which show Hattie combines studies with totally different definitions of the influence in question, Hattie used this defense,

"A common criticism is that it combines 'apples with oranges' and such combining of many seemingly disparate studies is fraught with difficulties. It is the case, however, that in the study of fruit nothing else is sensible" (VL, p. 10).

This defense does not deal with the specific details, e.g., in the category of Feedback, Hattie combines Standley (1996) which is about background music on production lines, with Rummel & Feinberg (1988) which is about giving students money and sweets as rewards, as if they were about about the same thing, i.e., feedback to students in a classroom. In addition, Hattie mistakenly reports the ES from Rummel & Feinberg (1988) as +0.60 rather than -0.60. 

Hattie never gives specific defenses to these.

Strategy 3 - Hattie can often agree with the critique saying, "Yes you must be sensitive to that" but then not address the issue at all, e.g., a common response from the peer reviewers, e.g., Prof Scott Eacott (2018),

 "Disappointingly, Hattie's response was in my opinion, inadequate" (p. 4).

Strategy 4 - Quietly remove the problem studies, e.g., In Wisniewski, Zierer & Hattie (2020) "The Power of Feedback Revisited", Hattie indirectly (&quietly) admits to the substance of these critiques by removing 8 of the original 23 meta-analyses & and drastically reducing the power of another 11 of the original 23 meta-analyses in his category of Feedback. The resultant effect size has then reduced from 0.79 Hattie & Timperley (2007) to 0.73 Visible learning (2009) now down to 0.48 Wisniewski, Zierer & Hattie (2020).

Strategy 5 - Change the focus of VL from the ES numbers to the "Story" narrative, e.g., with Lovell, Larsen & Kraft, when questioned about the fundamental premise of Visible Learning that the ES determines "what works best", Hattie backtracks, e.g.,

"And they look at that effect-size table and say tick, tick, tick to the top influences and no, no, no to the bottom, and this was never my message." (Hattie in Hattie & Larsen, 2020, p. 28).

Typical Responses to Hattie's Defenses

Snook, Clark, Harker, O’Neill & O’Neill (2010) respond to Hattie's 2010 defense,

"In our view, John Hattie’s article has not satisfactorily addressed the concerns we raised about the use of meta-analyses to guide educational policy and practice." (p. 97)

Prof Topphol responds to Hattie's defense,

"Hattie has now given a response to the criticism I made. What he writes in his comment makes me even more worried, rather than reassured."

Eacott (2018),

"Hattie did produce an antithesis to my original paper. His response made a counter argument to my claims. To do so, he did not need to engage with my ideas on anything but a superficial level (to think with Ryle, this is the wink all over – a focus on a few words without grasping their underlying generative meaning). There was no refutation of my claims. There was no testing of my ideas to destruction and no public accountability for his analysis. If anything, he simply dismissed my ideas with minimal reference to any evidence" (p. 6).

Eacott (2018) goes further and talks of Hattie systematically NOT addressing the counter claims and critiques of many scholars. 

"Hattie seeks to tell teachers what to do on the basis of what is presented as robust scientific evidence (irrespective of its critiques and failure to systemically refute counter claims)" (p. 6).

Simpson (2019b) highlights how some of the defences follow a familiar pattern:

"listing assumptions which are not checked (but are often prima facie absurd), arguing that the conclusions from their flawed arguments should stand until they are proved wrong, or claiming that mere awareness of the flaws is sufficient for them to continue with the same arguments. At worst, they fall back on arguing that they know about the criticisms, so can now exclude them from further consideration." (p. 4-5)

Lovell also posted a detailed review of Hattie's answers here and summarises:

"I came to the conclusion that combining effect sizes from multiple studies, then using these aggregated effect sizes to try to determine ‘what works best’, equates to a category error... I asked myself the following, ‘Has an effect size ever helped me to be a better teacher?’ I honestly couldn’t think of an example that would enable me to answer ‘yes’ to this question... But if for you, like me, the answer is ‘no’, then let’s agree to stop basing policy decisions, teacher professional development, or anything else in education upon comparisons of effect sizes. As both John and Adrian suggest, let’s focus on stories and mechanisms from now on."

Steven Kolber writes an insightful blog on this,

"I lean towards the argument refutation approach as it involves actively engaging with another person’s ideas and responding to them. This has consistently been seen as something that Hattie avoids, instead rolling out similarly phrased platitudes without directly addressing those arguments, or more commonly refutations, in opposition to his work."

Darcy Moore posts,

"Hattie’s [2017] reply to Eacott’s paper does not even remotely grapple with the issues raised."

Prof Eacott also responded to Hattie's defense,

"Disappointed that SLAM declined my offer to write a response to Hattie's reply to my paper. Dialogue & debate is not encouraged/supported."

Eacott (2018) was able to publish a response in a different journal,

 "Disappointingly, Hattie's response was in my opinion, inadequate" (p. 4).

"given my argument for his work being Tayloristic (and supporting evidence), in what ways is his work beyond that of Taylor? Are there no commercial arrangements with ACEL? Is his work not highly influential in policy discussions despite questioning of the very foundations of his analysis? Is his name not deployed by politicians, systemic authorities and school leaders as an authority/authoritative source? If anything, what Hattie has inadvertently done is support my argument while attempting to refute it" (p. 6).

Bakker, Cai, English, Kaiser, Mesa & Van Dooren (2019) confirm Hattie's approach is to ignore critique,

"...his lists of effect sizes ignore these points and are therefore misleading."

Professor Bergeron in his voicEd interview also talks about Hattie's conflict of interest and Hattie's reluctance to address the details of his critics. Listen here - at 17min 46sec.

In addition, Eacott (2018) discusses the "uncritical adoption" of Hattie by policy makers and schools as no dissenters or counter arguments are allowed (p. 6).

Prof Dylan Wiliam casts significant doubt on Hattie's entire model by arguing that the age of the students and the time over which each study runs is an important component contributing to the effect size. 

Supporting Prof Wiliam's contention is the massive data collected to construct the United States Department of Education effect size benchmarks. These show a huge variation in effect sizes from younger to older students. 

This demonstrates that age is a HUGE confounding variable or moderator since, in order to compare effect sizes, studies need to control for the age of the students and the time over which the study ran. Otherwise, differences in effect size can be due to the age of the students measured!

Given Hattie's conclusion in his 2015 defense,

"The main message remains, be cautious, interpret in light of the evidence, search for moderators, take care in developing stories, welcome critique..." (p. 8)

I'm extremely surprised Hattie has not addressed the massive implication of this evidence to his work, all he says in his summary VL 2012 (p. 14),

"the effects for each year were greater in younger and lower in older grades ... we may need to expect more from younger grades (d > 0.60) than for older grades (d > 0.30)."

Hattie finally agrees (2015 defense) with Prof Wiliam:

"Yes, the time over which any intervention is conducted can matter (we find that calculations over less than 10-12 weeks can be unstable, the time is too short to engender change, and you end up doing too much assessment relative to teaching). These are critical moderators of the overall effect-sizes and any use of hinge = 0.4 should, of course, take these into account." (p. 3)

Yet Hattie DOES NOT take this into account, there has been no attempt to detail and report the time over which the studies ran nor the age group of the students in the question nor adjust his previous rankings or conclusions.

Professor Dylan Wiliam summarises, 

"the effect sizes proposed by Hattie are, at least in the context of schooling, just plain wrong. Anyone who thinks they can generate an effect size on student learning in secondary schools above 0.5 is talking nonsense."

The U.S Education Dept benchmark effect sizes support Wiliam's contention.

The Education Endowment Foundation & Evidence 4 Learning give there defense to Prof Adrian Simpson's critique here.

In addition, Eacott discusses the "uncritical adoption" of Hattie by policy makers and schools as no dissenters or counter arguments are allowed (p. 6).