Effect Size d= 0.73 (Hattie's Rank=10).

A short video summary of the arguments on this page - here.

The 23 meta-analyses Hattie cities are:

While all these studies are GENERALLY about FEEDBACK, they totally differ on the specific aspect of feedback. For example, they range from extrinsic vs intrinsic rewards (money & chocolate vs pleasure & achievement), background music, feedback to the teacher vs to the student, computer feedback, timing of feedback, etc.

These studies highlight the major problem with Hattie's (& the EEF's) method of the combining disparate studies to get ONE effect size.

Hattie acknowledges this and his defense is,
"A common criticism is that it combines 'apples with oranges' and such combining of many seemingly disparate studies is fraught with difficulties. It is the case, however, that in the study of fruit nothing else is sensible" (VL, p. 10).
But, Hattie's response cleverly avoids the need to justify specific studies, e.g., why is Standley (1996) included when it is about background music and not about the sorts of feedback strategies used by teachers?

Greg Ashman (2018c) also details similar problems with the EEF's top strategy, meta-cognition, which Ashman says,
"appears to be a chimera; a monster stitched together from quite disparate things."
Sundar & Agarwal (2021) analyse Hattie and also warn of this, 
"If learning strategies included in the meta-analysis are not consistent and logical, then beware! For example, if you find a meta-analysis that groups together “feedback” strategies including teacher praise, computer instruction, oral negative feedback, timing of feedback, and music as reinforcement, does that sound consistent to you?" (p. 6)
Ekecrantz (2015), Mannion (2017), Mannion (2020) & Fletcher-Wood (2021) also cast doubt about the EEF's & Hattie's representation of Feedback. It was interesting that in a webinar defending the EEF, Steve Higgins (2019) borrows Hattie's 'fruit' response to defend the combining of disparate studies.

Hattie in a recent webinar muddies the water further by claiming, 
"I'm not even sure there is a concept such as formative or summative assessment." (@ 30minutes)
The details in the peer review summarised in this blog, show over & over again, that the way Hattie defines, selects & groups studies is a highly questionable.

Dylan Wiliam had a humorous response to this issue,
"This underscores the importance of adequate theorization, identified by Kelley (1927). The "jingle fallacy" is assuming that two things with the same name are in fact the same, while the "jangle fallacy" is assuming that things with different names, are, in fact different."
Hattie & the EEF Obscure the True Nature of Feedback

It is ironic that one of the studies that Hattie used, Rummel & Feinberg (1988), warned of the major problem of combining disparate studies as Hattie & the EEF continue to do,
"It is argued that by including studies that claim to examine this theory but are in reality, not adequately operationalizing the theoretical propositions, this only serves to obscure the true nature of this area" (p. 160).
Prof Terry Wrigley (2015) in Bullying by Numbers, gives an English humorist critique of Hattie's & the EEF's method,
"Its method is based on stirring together hundreds of meta-analyses reporting on many thousands of pieces of research to measure the effectiveness of interventions.  
This is like claiming that a hammer is the best way to crack a nut, but without distinguishing between coconuts and peanuts, or saying whether the experiment used a sledgehammer or the inflatable plastic one that you won at the fair" (p. 5).
Hattie Finally Concedes

More detail of the peer review critique on Hattie's use of disparate feedback studies is below and it appears Hattie has finally conceded & did not refer to his usual apples vs oranges = fruit response. 

In Wisniewski, Zierer & Hattie (2020) "The Power of Feedback Revisited", Hattie admits these differences, or heterogeneity of definitions, is a major problem.
"...the significant heterogeneity in the data shows that feedback cannot be understood as a single consistent form of treatment." (p. 1)

While not directly acknowledging the wide variety of issues in VL, Hattie and his co-authors completely excluded 8 of the original 23 meta-analyses and partially excluded a further 11 meta-analyses, resulting in a substantially reduced ES of 0.48 - details in Feedback Revisited.

More Detail on the Nature of these Feedback Studies

Ruiz-Primo & Li (2013) Examining formative feedback in the classroom context: New research perspectives, reviewed over 9,000 studies on feedback (most of the studies Hattie used) and decided only 238 were of high enough quality to use (p. 217). But they also warn that,

"only 131 studies, or 4%, were considered appropriate for reaching some type of valid conclusion based on their selection criteria" (p. 218).
They detail that studies differed in many respects (p. 217):

They conclude,
"Clearly, the range of feedback definitions is wide... how is it possible to argue for feedback effects without considering the nuances and differences among the studies?" (p. 217).
Many other scholars also show Hattie just combines all these different studies together without regard for their differences, e.g.,

Proulx (2017) in Critical Essay on the Work of John Hattie for the Teaching of Mathematics, observes that, Hattie's definition of feedback is not consistent with the collection of feedback studies he cites and therefore has nothing to do with his aim of 'what works best'. 

Proulx also expressed his concern that even though Hattie compares studies with totally different definitions of feedback, Hattie constantly reminded us 'feedback' is one of the best approaches to teaching.

Nielsen & Klitmøller (2017) in Blind spots in Visible Learning - Critical comments on the "Hattie revolution", discuss in detail the many problems of Hattie's synthesis of feedback studies. They start with  Hattie's definition of feedback,
"... feedback is information provided by an agent (e.g., teacher, peer, book, parent, or one’s own experience) about aspects of one’s performance or understanding. For example, a teacher or parent can provide corrective information, a peer can provide an alternative strategy, a book can provide information to clarify ideas, a parent can provide encouragement, and a learner can look up the answer to evaluate the correctness of a response. Feedback is a 'consequence' of performance" (VL, p. 174).
"In summary, feedback is what happens second, is one of the most powerful influences on learning, occurs too rarely..." (VL, p. 178). 
They then detail a significant problem with Hattie's work in general but with the influence of feedback in particular, i.e., the different definitions of variables,
"it is our assessment that in four of the five 'heaviest' surveys that mentioned in connection with Hattie's cover of feedback, it is conceptually unclear whether they are operates with a feedback term that is identical with Hattie's" (p. 11, translated from Danish).
Furthermore, they state (p. 10),
"The breadth of the phenomenon of feedback varies clearly in the meta-analyses used."
They then go into more detail,
"... we will come closer to look at five of the meta-analyses that Hattie builds his calculation on... Hattie's feedback area consists of 23 meta-analyses including 67,931 people, 5 of them are a special heavy because they include 62,761 people corresponding to 92% of the total sample" (p. 11).
This also shows the major issue of how to weight studies and how different weightings derive totally different effect sizes.

They define their criteria for examination of the studies (p. 11):

1. Are the surveys valid, do they measure what Hattie says, i.e., feedback?

2. Are the meta-analyses transparent, so it is possible to examine the quality of the individual studies in the meta-analyses?

3. The use of randomized control group studies? Studies which use control groups are of higher quality.

They conclude,
"...our analysis shows that the transparency is low in two out of five studies, also only one of the five studies is consistently working with a control group design.  
...the study by Kluger and DeNisi (1996), that Hattie (VL, p175) denotes 'the most systematic study addressing the effects of various types of feedback' has an effect size of d = 0.38 - i.e., a much lower impact assessment than the 0.73 ...other than that 38 percent of the surveys that are included in Kluger and DeNisi's study, a negative effect on the learning process - which is moreover contrary to Hattie's assumption that 'almost everything works'. 
Kluger and DeNisi (1996) therefore denote feedback as a two-fold sword that both can lead to the student either learning significantly more or significantly less" (p. 11).
One of their pertinent observations is that many of the studies produce negative effects. They quote Shute (2008) 'Within this large body of feedback research, there are many conflicting findings and no consistent pattern of results.' 

Barwe and Dahlström (2013) also argue it is better to look at a good meta-analysis rather than average a whole bunch of disparate meta-analyses as Hattie does.
"If one would have to choose between Hattie's average measure of the effect size of a certain influencing factor or the measure that a high-quality meta-analysis presents, we recommend the later. For example, Hattie takes up a systematic, high-quality meta-study of Kluger & DeNisi (1996) which deals with feedback and where the effect size is = 0.38 (p. 175).
Maybe you can have more confidence in that study and its value than the average Hattie produces (= 0.73)." (p. 27).
The Best Feedback Study - Kluger & DeNisi (1996)?

Hattie & the EEF claim the highest quality study is Kluger & DeNisi (1996) with an effect size of 0.38, which is below Hattie's magic hinge point of 0.40.

Also, this is about half of the overall effect size for Feedback of 0.73. 

Given Hattie's comments about weighting previously, why is this study not weighted higher?

David Didau (2016) briefly summarises the Kluger and DeNisi study (listen around 24 mins).

Professor Dylan Wiliam confirms over 38% of studies show feedback has a negative effect (see video below).

Note: Hattie under reports this figure as 32% (VL, p. 175).

Busch & Watson (2019). The Science of Learning: 77 Studies That Every Teacher Needs to Know. Also detail that, the negative result of feedback, in this study, is one of the most important findings in the research (go to 1hr, 35 mins).

So there is strong counter to Hattie's disparaging claim that,
"When teachers claim that they are having a positive effect on achievement or when a policy improves achievement, this is almost always a trivial claim: Virtually everything works.  
One only needs a pulse and we can improve achievement" (VL, p. 16).
Prof Wiliam details more problems with the feedback research - most of the studies are on university students and 85% of the feedback is ONE event lasting minutes!!! He then goes further and says that if you compare these different studies on feedback (as Hattie does),
"Your results are meaningless."

Yet Hattie says, 95-97% of things we do enhances achievement!

But There are Doubts about Kluger & DeNisi (1996)

Fletcher-Wood (2021) in Research: Is all our evidence all it’s cracked up to be? casts doubt on this studies' relevance to school age students, as most of the studies are on adults and university students.
"First, Kluger and DeNisi focused on the way feedback affects behaviour – not how it affects learning. Later authors extended Kluger and DeNisi’s conclusions to argue that feedback has powerful effects on learning – but this isn’t fully justified by the original research.  

Second, Kluger and DeNisi included a range of studies – including those testing the effect of feedback on workers’ use of ear protection, hockey players’ body checks, and people’s extra-sensory perception (apparently feedback helps). Only nineteen of the 131 studies included were in schools and most focused on changing classroom behaviour – not learning."
So once again we have the 'apples vs oranges' problem and also another problem we have mentioned many times - see Student Achievement, of Hattie mixing & combining studies that do not measure achievement but something else, in this case behaviour.

Rummel & Feinberg (1988) - Details

Professor Robert Slavin (2018) in his blog John Hattie is Wrong, gives a pertinent example of Hattie's use of feedback studies,
"A meta-analysis by Rummel and Feinberg (1988), with a reported effect size of +0.60, is perhaps the most humorous inclusion in the Hattie & Timperley (2007) meta-meta-analysis. It consists entirely of brief lab studies of the degree to which being paid or otherwise reinforced for engaging in an activity that was already intrinsically motivating would reduce subjects’ later participation in that activity. Rummel & Feinberg (1988) reported a positive effect size if subjects later did less of the activity they were paid to do. The reviewers decided to code studies positively if their findings corresponded to the theory (i.e., that feedback and reinforcement reduce later participation in previously favored activities), but in fact their “positive” effect size of +0.60 indicates a negative effect of feedback on performance. 
I could go on (and on), but I think you get the point. Hattie’s meta-meta-analyses grab big numbers from meta-analyses of all kinds with little regard to the meaning or quality of the original studies, or of the meta-analyses."
Janson (2018) also confirms this,
"the positive effect size found indicates a negative effect of that feedback on the performance of that activity."
Some studies that Rummel and Feinberg (1988) used are shown below and Prof Slavin is correct,

The regulatory body in mine and Hattie's jurisdiction, The Victorian Institute of Teaching (2018), publish dismissal proceedings for teachers they deem unfit to teach.
"The evidence of possible serious misconduct or lack of fitness to teach ... was: 
 Inappropriate gift buying, described by students as bribery, such as sweets and other items."
Standley (1996) - Music as Feedback? The highest effect size of 2.87

Schulmeister & Loviscach (2014) in Critical comments on the study "Making learning visible" (Visible Learning) also detail many problems with Hattie's analysis of feedback. The problem of averaging many different studies on different target groups (teachers and students) using very different feedback mechanisms. For example, the Standley (1996) study is about the impact of music on behavioral interventions. They conclude,

"Only in a very broad sense has this study something to do with feedback; it is behavioristic reinforcement" (p. 9).
I checked the Standley (1996) and their results were interesting to me as I managed a band that played in pubs in Melbourne in the 1990's and the debate was about live music versus DJs or recorded music. But Standley (1996, p. 108) answered that question for us,
"Live music (ES = 1.13) was more effective than recorded music (ES = 0.86)."

Prof Terry Wrigley (2015) in Bullying by Numbers, critiquing the EEF in particular but also Hattie,
"Specifically, on Feedback, the Toolkit provides some more specific references to back up its very general claims, but many of these are over 20 years old and currently unobtainable. Seven more detailed references are given, each with an ‘effect size’, but these range from .97 to .20. Which is to be believed? Summaries follow, in highly technical language, mostly without indicating which stage or subject, what kind of learning, what kind of feedback, which countries the research took place in, and so on. Some of the sources are very critical of particular types of feedback... 
Meta-analyses are used in Medicine to enable researchers to complement the reading of other research, though not to substitute for it; for example, if experiments have been based on small samples, averaging the results can suggest a general trend. 
But the medical literature contains serious warnings against the misuse of meta-analysis. Statisticians are warned not to mix together different treatments, types of patient or outcome measures – the ‘apples and pears’ problem. If the original results differ strongly, they are advised to highlight the difference, not provide a misleading average. This is exactly what has not happened in the Toolkit, which should never have provided an average score for “Feedback” since the word has so many meanings" (p. 6).
Hattie's Latest Defense

In Real Gold Vs Fool's Gold (2020), Hattie continues with this 'fruit' response and once again does not address the significant issues of disparate studies,
"Any literature review involves making balanced judgements about diverse studies. A major reason for the development of meta- analysis was to find a more systematic way to join studies, in a similar way that apples and oranges can make fruit salad. Meta-analysis can be considered to ask about “fruit” and then assess the implications of combining apples and oranges, and the appropriate weighting of this combination." (p. 3)

But, lack of appropriate weighting of high quality studies is another issue Hattie has been criticized for - see Effect Size -Problem 8.

 Hattie (2020) continues,

"Unlike traditional reviews, meta-analyses provide systematic methods to evaluate the quality of combinations, allow for evaluation of various moderators, and provide excellent data for others to replicate or recombine the results. The key in all cases is the quality of the interpretation of the combined analyses. Further, as noted above, the individual studies can be evaluated for methodological quality." (p. 4)

Once again, Hattie cleverly implies he has addressed all these issues, but the peer review examples above show this is not the case.

Note: Hattie has recently removed Standley (1996) & Rummel & Feinberg (1988) without any admission of error or any rationale at all!

The Problem With Most Feedback Studies

Ruiz-Primo & Li (2013) detail the quality issues which we consistently see in Hattie's synthesis,

"A high percentage of papers investigating the impact of feedback did so without using a control group... 
Confounded effects, rarely mentioned in the synthesis and meta-analyses, pose another threat to validity when interpreting results of feedback studies" (p. 218).
"most of the studies do not provide information about the reliability and validity of the instruments used to measure the effects of feedback on the selected outcomes. The validity of feedback studies is threatened by a failure to attend to the technical characteristics of the instruments used to measure learning outcomes... Given these measures with ambiguity in technical soundness, can we fully trust results reported in synthesis and meta-analysis studies?
... there is an issue of ecological validity. For a research study to possess ecological validity and its results to be generalizable, the methods, materials, and setting of the study must sufficiently approximate the real-life situation that is under investigation. Most of the studies reported are laboratory-based or are conducted in classrooms but under artificial conditions (e.g., students were asked to identify unfamiliar uses of familiar objects)...
Furthermore, a high percentage of the studies focus on written feedback, and only a few on oral or other types of feedback, although oral feedback is more frequently observed in teachers’ daily assessment practices (see Hargreaves et al., 2000).
...we argue that formative feedback, when studied in the classroom context, is far more complex than it tends to appear in most studies, syntheses, or meta-analyses. Feedback practice is more than simply giving students feedback orally or in written form with externally or self-generated information and descriptive comments. We argue that feedback that is not used by students to move their learning forward is not formative feedback. We thus suggest that feedback needs to be examined more closely in the classroom setting, which should ultimately contribute to an expanded and more accurate and precise definition" (p. 219).
Some interesting findings,
"Research has made clear that students hardly read teachers’ written feedback or know how to interpret it (Cowie, 2005a, 2005b)" (p. 225).
"most of the publications on formative assessment and feedback include examples of strategies and techniques that teachers can use. Most of them, however, do not provide empirical evidence of the impact of these strategies on student learning; nor do they link them to contextual issues that may affect the effectiveness of the strategies...
there is a lack of studies conducted in real classrooms—the natural setting—where it would be important to see evidence that feedback strategies have substantive impact. Moreover, few studies have focused on feedback over extended periods or on factors that can moderate or mediate the effectiveness of feedback. Therefore, we cannot generalize what we know from the literature to classroom practices...
Rather than persisting with our common belief that feedback is something doable for teachers, we should strive to study formative assessment practices in the classroom, including feedback, to help teachers and students to do better. Given these unanswered questions, we need different and more trustworthy strategies of inquiry to acquire firsthand knowledge about feedback in the classroom context and to systematically study its effects on student learning" (p. 226).
The Focus of Feedback in English Schools:

The poor research studies used as the basis for deciding that "Feedback" is a high impact study may account for the lack of success of focusing on it as an initiative.

Wiliam (2019) notes,
"the EEF’s emphasis on feedback as the single most cost-effective intervention justified 'additional pressures on teachers from inspectors that are ultimately not productive' even though few, if any, of the studies that the EEF included in its review looked at the effects of marking in school."
Cohen (2019) also comments about this lack of success quoting Christodoulou (2016),
"drawing on similar foundational assumptions, takes on the issue of feedback in the context of the English school system. She deals with the puzzling failure of Assessment for Learning. AfL was a government programme for rolling out feedback strategy based on strong evidence from a range of scholarly sources, including experimental evidence. It commanded strong support among policymakers and a great deal of the teaching profession. It was successfully implemented at least to the extent that teachers in England now provide a great deal more feedback than before, and more than teachers in other countries. Yet, the theorised improved student outcomes did not materialise. 
Christodoulou (2016) uses a range of evidence to argue that part of the reason for this is a failure to differentiate formative and summative assessment."
Feedback is True but Useless

Christodoulou & Ashman discuss Wiliam's famous quote, 'feedback is true but useless' in their podcast here @ 18 minutes.

They talk about Wiliam's observation of a science class who hand in a report and the teacher provides written feedback, "you need to be more systematic in your investigation."

Wiliam then asks the student what are you going to do next?

The student replied, "I don't know, if I'd know how to be more systematic I would have done that."

They continue with other examples like telling a comedian who is not funny, you need to be more funny.

Christodoulou then recommends for a student to improve a certain skill they need to be given an activity that they can repeat and practice.

Work Load & Feedback:

Glenn Pearsall (2017), one of the most popular teaching experts in Australia links teacher work load with inefficient feedback practices, in his TER podcast, he said,
"Great feedback which the kid does not act on is a waste of both the kid and the teacher's time!"
Internal Feedback Using Comparisons

Prof David Nicol has done some exciting work involving students using comparisons & peer reviewing to get internal feedback about their work. Nicol claims this is more powerful than external feedback.

Nicol talks about this being a more natural process which is highly engaging. With the added benefit of reducing Teacher workload.

PISA and Feedback

David Didau (2017b) in his excellent blog on feedback, looks at the key studies used by Hattie and the EEF and confirms Wiliam's analysis and also shows feedback is complicated and nuanced.

He also pointed out that even though most researchers have feedback as an important teaching strategy, PISA has feedback NEGATIVELY correlated with Science performance.

The Negative Influences

From PISA 2015 Volume 2 (page 228).

Price, Handley, Millar & O'Donovan (2010). Feedback: all that effort, but what is the effect? Confirm feedback is complex and add relationships are important, e.g.,
"Measuring ‘effectiveness’ requires clarity about the purpose of feedback. Unless it is clear what feedback is trying to achieve, its success cannot be judged... 
Although a frequently used term, feedback does not have clarity of meaning. It is a generic term which disguises multiple purposes which are often not explicitly acknowledged. The roles attributed to feedback fall broadly into five, but not entirely delineated discrete, categories: correction, reinforcement, forensic diagnosis, bench-marking and longitudinal development (feed-forward)... 
Accurate measurement of feedback effectiveness is difficult and perhaps impossible. Furthermore, the attempt to measure effectiveness using simple indicators – such as input measures or levels-of-service – runs the risk of producing information which is misleading or invalid and which may lead to inappropriate policy recommendations."
Prof Paul Kirschner (2018) details the problems with feedback.

Michael Pershan (2019) writes an excellent and insightful blog on feedback. He basic argument is the evidence is poor and the notion of feedback is too general to be of any help to teachers.

Buckingham & Goodall (2019) in the Harvard Business Review - The Feedback Fallacy, detail arguments that feedback is a complex phenomena.

Hattie's Corwin presentation where he asks what is Feedback?

Hattie seems to change his definition of feedback to "where to next". Later he also says the most important aspect of feedback is the teacher receiving feedback.

The problem with feedback in John Hattie's Visible Learning - an analysis of Hattie's 2020 study on feedback revisited -

Prof Richard E. Clark on the Kluger & Denisi study here @ 49 minutes.

1 comment:

  1. thanks for your gr8 work, Hattie must be taking notice, he just removed the Rummell and Standley study. But, he has a lot more studies to remove!