The Story?

Is it "The Story" or "The Numbers"?

Patrick O'Connor (2020) wrote insightfully about Hattie's switch to "The Story" narrative,

"As Hattie’s statistics have come under greater scrutiny, his emphasis on the narrative value of Visible Learning has increased. He and Hamilton (2018, p. 46) insisted that ‘the key here is that it is the INTERPRETATION and STORY that help explain the findings that is a contender for a useful theory; not the data’.  

This emphasis is peculiar given that meta-analysis emerged with the promise that it offered, in the words of Gene Glass (1976, p. 3), who coined the term, ‘a rigorous alternative to the casual, narrative discussions of research studies’. Hattie’s remarks raise the question as to what exactly is the connection between the Visible Learning ‘story’ and education research data." (p. 142)

An example of Hattie's switch to "The Story" was in an interview with Lovell, who questions Hattie about using effect sizes & rankings to determine "what works best",

"that's too simplistic, I would never do that..."

"What's the story, not what's the numbers..."

"that’s why this will keep me in business to keep telling the story..." (@46minutes)

"it's the story, it's the story, its the story, but... look for evidence you are wrong." (@95minutes 30sec)

Similarly with Matthew Kraft, Hattie states,

"the greatest was the misuse of the ranking of the different influences on student achievement. I included the ranking as an appendix in the book at the last minute, but many saw it as the major story. In reality, each of the (now 300) influences is not unique... 

But since 2009, I have tried to discourage focusing only on the top influences and ignoring the lower effects..."

Kraft responds,

"That is fascinating that your list of factors related to student achievement was a last-minute addition but has become arguably the most influential part of your book." (Kraft & Hattie , 2021)

However, if you look at Hattie's 2005 ACER presentation, Hattie's rankings were firmly in place at least 3 years before Visible Learning was published in 2008.

Hattie also used this "last minute" excuse to explain the significant blunder he made with the statistic, Common Language Effect Size (CLE) - see here.

"at the last minute in editing I substituted the wrong column of data into the CLE column and did not pick up this error; I regret this omission." Hattie (2015)

Then again With Larsen, Hattie retreats from his rankings,

"Too often, this ranking has been misinterpreted with some saying these top ranked are good, these lower ranked are bad." (Larsen & Hattie, 2020, p. 18) 

"And they look at that effect-size table and say tick, tick, tick to the top influences and no, no, no to the bottom, and this was never my message." (ibid, p. 28) 

"Too many readers thought I was saying we can rank all effects one by one (whereas there are underlying themes differentiating the top and bottom influences), too many thought 0.40 was somehow magic." (ibid, p. 79)

Then in a specific example of Hattie's representation of the Feedback research, Hattie & his coauthors remove most of the original 23 studies cited in VL - see Wisniewski, Zierer & Hattie (2020), "The Power of Feedback Revisited" and admit to flaws in Hattie's method of using one average ES,

"Care is needed, however, with focusing too much on the average effect size as the integrated studies show high heterogeneity, suggesting that, conform to expectations, not all feedback is the same and one (average) effect size does not fit all." (p. 12)

BUT - Hattie's Presentation Changes Depending on Who He talks to

As noted above, Hattie completely changes the narrative when talking to academics who are aware of the huge critique of his work.

However, at the same time, in webinars largely to people who are unaware of this critique, Hattie reverts back to the simplistic notions of rankings and that somehow 0.40 is magical, e.g., with Corwin Mind Frames webinar in 2019 (@ 7minutes), Hattie presents slides with a clear "NO" on influences with low ES, says "they don't matter much" and also he clearly promotes an ES=0.40 as magical, the point at which 1 years growth is achieved - 

In addition, Hattie's latest book, Hattie & Hattie (2022), 10 Steps to Develop Great Learners: Visible Learning for Parents, Hattie & his son Kyle reemphasise the claims of VL, that his rankings using the ES determines "What Works Best" and that Systemic Factors "don't matter much", e.g., page 156-

Details of Hattie's recent webinars (2018-2022) where he contradicts what he claimed to Lovell, Kraft & Larsen are here.

What's "The Story"?

The original story of Visible Learning was based on ONE number the ES. Hattie claimed most studies published have +ve ES, therefore "everything works", so we should change our focus from "what works" to "what works best" and his rankings were a logical consequence of this - NOT a last minute thought as he claims.

However, as stated above, with academics, Hattie retreats from the claim that the ES determines "what works best" and switches to a clever and ambiguous narrative "The Story". 

However, it is not entirely clear what Hattie means by "The Story".

At times, it is about his simplistic rankings described in a different more general way - The "blue" zone (ES>0.40) versus the "yellow" zone (ES<0.40).

At other times it is about why certain influences have low ES, e.g., 

"The way I see the evidence and my interpretation of it is that the evidence so far shows that knowledge of the subject, the pedagogical content knowledge, whatever you want to call it, hardly matters, which is why we’ve spent the last 10 or 12 years trying to understand why it doesn’t matter – because it should matter." (Larsen & Hattie, 2020, p. 16)

Although, Hattie has not tried too hard to find explanations for these low effect sizes, e.g. look at Prof Robert Coe's explanation in more detail about subject matter knowledge.

Then again, at other times, it is about the detail of an influence, e.g., Feedback. Hattie claims the main aspect of feedback is "where to next", even though NONE of the meta-analyses he cites define this notion let alone measures it.

Strangely, in contradiction with "the story" narrative, Hattie starts Visible Learning (VL) with this aim in his preface, 

"It is not a book about classroom life, and does not speak to the nuances and details of what happens within classrooms."

NOT the Teacher's or other Researcher's Stories!

In an interview with Hanne Knudsen (2017), Hattie stated,

"almost every teacher wants to get up and talk about their story, their anecdotes and their classrooms. We will not allow that, because as soon as you allow that, you legitimise every teacher in the room talking about their war stories, their views, their kids" (p. 3).

Criticisms of VL from classroom teachers was that VL often contradicted their classroom experience & stories and failed to account for nuance, context and complexity of the classroom e.g., class size, subject matter knowledge, welfare policies, etc.

Lead "classroom size" researchers, Prof Peter Blatchford and David Zyngier have emphasised the need to use different study designs, such as longitudinal studies, rather than a sole focus on effect size & meta-analyses in order to discover complex stories and interactions, particularly in the class size debate.

Their evidence suggests that class size is important in a number of complex ways, but Hattie dismisses their "story" using his trump card "the effect size" - see class size.

One Average Effect Size Loses The Story

Hattie's method of using one average to represent many meta-analyses loses the story of each individual meta-analysis - e.g., class size.

Ironically, the author of the largest class size study, Gene Glass, who invented the meta-analysis methodology and who Hattie cites as the person who motivated him to use this method said, 

"The result of a meta-analysis should never be an average; it should be a graph."(Robinson, 2004, p. 29).

Thibault (2017) also questions Hattie's method of using one average to represent a range of studies (translation to English),

"We are entitled to wonder about the representativeness of such results: by wanting to measure an overall effect for subgroups with various characteristics, this effect does not faithfully represent any of the subgroups that it encompasses! 

...by combining all the data as well as the particular context that is associated with each study, we eliminate the specificity of each context, which for many give meaning to the study itself!"

This problem was raised with Hattie over 10 years ago,

"The methodology involved in meta-analyses often obscures rather than illuminates what is needed for good policy decisions. The grouping of various factors into one “effect size” often hides the complexity of the issues...

The results reported in meta-analyses of educational research tend to mask complex interactions between schools, students, families and communities. Without this awareness, lists of “effect sizes” merely encourage silver bullet responses to complex educational problems." (Snook et al., 2010, p. 97)

But The Problem of Valuing One Average ES Continues

Hattie's simplistic notion of comparing ONE average ES for systemic vs teacher effects has already gained popularity amongst politicians and leaders, and they quickly promoted VL, as Eacott (2017) accurately observed,

"Hattie’s work has provided school leaders with data that APPEAL to their administrative pursuits."

The Background Studies

When I've read the background studies I've often found Hattie either combines studies that DON'T have a consistent definition of the influence, e.g., Feedback, or Hattie reports the opposite result of those studies. A clear example is class size, e.g., Barwe and Dahlström (2013),

"here we have three meta-studies that show a strong connection between class reduction and student achievement, but in Hattie's synthesis this conclusion does not emerge. That's remarkable." (translated p. 22)

Also, Gene Glass the author of the largest class-size study (see - class size) that Hattie cites responded,

"Averaging class size reduction effects over a range of reductions makes no sense to me. It's the curve that counts."

Hattie responded to my critique, in Hattie & Hamilton (2020, p. 12)

Yet, in the same publication Hattie & Hamilton are contradictory about what a "small" ES means (p. 2),

Hattie's "story" about a small ES does not go into the detail of the research that shows systemic aspects like class size are measure by standardised tests and these can yield effect sizes 1/5th less than that measured by a specific test (Ruiz-Primo et al. (2002)). This in part accounts for systemic influences having smaller effect sizes than specific influences, e.g., Class size vs Feedback.

Nor does Hattie deal with the critique that his averaging of ALL types of class size reductions e.g., 40 down to 30, 30 down to 25, 25 down to 15, etc., reduces the specific ES for a particular reduction e.g., 25 down to 15.

Nor does Hattie deal with the significant peer review critique of his averaging & comparing ES from disparate & poor quality studies. (Ashman (2022), Wiliam (2019), Wrigely (2015), Simpson (2018), Bergeron & Rivard (2017)).

Hattie's Propensity for Exaggeration in His "Story"

Examples include,

"...in my work I deal with class sizes of 1000, I know how to do that, I’ve got quite good at it over the last 40 years, how to teach to class of 1000." (Pedagogy Non-Grata, 2019, podcast @52minutes)

Yet, the Melbourne Graduate School of Education, where Hattie has been for last 11 years, mostly working with PhD students 1 to 1 has no room or theatre in this building that would fit anywhere near 1000 students.

I suspect what Hattie is referring to is his presentations once or twice a year to a general Education audience, e.g. ResearchEd Melbourne 2017, where there are less than 200 people in the audience. 

To equate this with the class size issue is misleading.

Then Hattie at, ResearchEd Melbourne (2017), referring to the studies he uses,

"nearly all of it is based on what happens in regular classrooms by regular teachers... 99.+% is based on classrooms run by ordinary teachers, not like in Psychology where they use under-graduate students, they bring in outsiders and this kinda stuff." (9minutes).

This is clearly wrong, as his paper The Power of Feedback Revisited (2020) shows him removing most of the 23 original studies on Feedback as they are not based on students in Educational settings.

Then again at ResearchEd Melbourne (2017), referring to Kambrya HS who he worked with in 2015, Hattie stated, Kambrya HS was in the bottom 10% & now in top 10% in the state on any measure. (@25minutes).

This is clearly wrong by looking up the test from the school during that time- see Revolution School.