Effect Size d= 0.73 (Hattie's Rank=10).

The studies Hattie cities in Visible learning:

Ruiz-Primo & Li (2013) Examining formative feedback in the classroom context: New research perspectives, reviewed over 9,000 studies on feedback (most of the studies Hattie used) and decided only 238 were of high enough quality to use (p217). But they also warn that,
'only 131 studies, or 4%, were considered appropriate for reaching some type of valid conclusion based on their selection criteria' (p218).
They detail that studies differed in many respects (p217):

They conclude,
'Clearly, the range of feedback definitions is wide... how is it possible to argue for feedback effects without considering the nuances and differences among the studies?' (p217).
Many other scholars also show Hattie just combines all these different studies together without regard for their differences, e.g.,

Nielsen & Klitmøller (2017) in Blind spots in Visible Learning - Critical comments on the "Hattie revolution", discuss in detail the many problems of Hattie's synthesis of feedback studies. They start with  Hattie's definition of feedback,
'... feedback is information provided by an agent (e.g., teacher, peer, book, parent, or one’s own experience) about aspects of one’s performance or understanding. For example, a teacher or parent can provide corrective information, a peer can provide an alternative strategy, a book can provide information to clarify ideas, a parent can provide encouragement, and a learner can look up the answer to evaluate the correctness of a response. Feedback is a “consequence” of performance' (VL, p174).
'In summary, feedback is what happens second, is one of the most powerful influences on learning, occurs too rarely...' (VL, p178). 
They then detail a significant problem with Hattie's work in general but with the influence of feedback in particular, i.e., the different definitions of variables,
'it is our assessment that in four of the five "heaviest" surveys that mentioned in connection with Hattie's cover of feedback, it is conceptually unclear whether they are operates with a feedback term that is identical with Hattie's' (p11, translated from Danish).
Furthermore, they state,
'The breadth of the phenomenon of feedback varies clearly in the meta-analyses used' (p10).
They then go into more detail,
'... we will come closer to look at five of the meta-analyses that Hattie builds his calculation on... Hattie's feedback area consists of 23 meta-analyses including 67,931 people, 5 of them are a special heavy because they include 62,761 people corresponding to 92 percent of the total sample' (p11).
This also shows the major issue of how to weight studies and how different weightings derive totally different effect sizes (see Effect Size).

They define their criteria for examination of the studies (p11):

1. Are the surveys valid, do they measure what Hattie says, i.e., feedback?

2. Are the meta-analyses transparent, so it is possible to examine the quality of the individual studies in the meta-analyses?

3. The use of randomized control group studies? Studies which use control groups are of higher quality.

AuthorValidityTransCon. Grps?
Lysakowski &Walberg, 1980Focus on reinforcement techniques.LowNo
This is not clearly defined.
Unclear relation to feedback.
Low validity.
Lysakowski &Walberg, 1982Focus on corrective feedback.HighYes
Unclear relation to feedback in educational situations.
Feedback in connection with test situation.
but not in connection with instruction and teaching.
Low validity.
Kluger and DeNisi, 1996Feedback intervention consistent with Hattie's definition.HighYes
Focus on feedback in educational situations.
High validity.
Witt, Wheeless & Allen, 2006Feedback for the teacher, not students.LowNO
Does the student benefit from the teacher receiving feedback.
Low validity.
Swanson & Lussier, 2001Focus on dynamic assessment.Highnot consistent
Unclear relation to feedback in educational situations.
Low validity.

They conclude,
'...our analysis shows that the transparency is low in two out of five studies, also only one of the five studies is consistently working with a control group design.  
... the study by Kluger and DeNisi (1996), that Hattie (VL, p175) denotes "the most systematic study addressing the effects of various types of feedback" has an effect size of d = 0.38 - i.e., a much lower impact assessment than the 0.73 ... other than that 32 percent of the surveys that are included in Kluger and DeNisi's study, a negative effect on the learning process - which is moreover contrary to Hattie's assumption that "almost everything works". 
Kluger and DeNisi (1996) therefore denote feedback as a two-fold sword that both can lead to the student either learning significantly more or significantly less' (p11).
One of their pertinent observations is that many of the studies produce negative effects. They quote Shute (2008) 'Within this large body of feedback research, there are many conflicting findings and no consistent pattern of results.' 

Professor Dylan Wiliam confirms this saying 40% of studies show feedback has a negative effect.

Wiliam details more problems with the feedback research - most of the studies are on university students and 85% of the feedback is ONE event lasting minutes!!! He then goes further and says that if you compare these different studies on feedback (as Hattie does),
'Your results are meaningless'

Professor Robert Slavin in his blog John Hattie is Wrong, gives a pertinent example of Hattie's use of feedback studies,
'A meta-analysis by Rummel and Feinberg (1988), with a reported effect size of +0.60, is perhaps the most humorous inclusion in the Hattie & Timperley (2007) meta-meta-analysis. It consists entirely of brief lab studies of the degree to which being paid or otherwise reinforced for engaging in an activity that was already intrinsically motivating would reduce subjects’ later participation in that activity. Rummel & Feinberg (1988) reported a positive effect size if subjects later did less of the activity they were paid to do. The reviewers decided to code studies positively if their findings corresponded to the theory (i.e., that feedback and reinforcement reduce later participation in previously favored activities), but in fact their “positive” effect size of +0.60 indicates a negative effect of feedback on performance. 
I could go on (and on), but I think you get the point. Hattie’s meta-meta-analyses grab big numbers from meta-analyses of all kinds with little regard to the meaning or quality of the original studies, or of the meta-analyses.'
I checked Rummel and Feinberg (1988) and Prof Slavin is correct, here's an example of some of the studies they used,

Schulmeister & Loviscach (2014) Critical comments on the study "Making learning visible" (Visible Learning) also detail many problems with Hattie's analysis of feedback. The problem of averaging many different studies on different target groups (teachers and students) using very different feedback mechanisms. For example, the Standley (1996) study is about the impact of music on behavioral interventions. They conclude,
'Only in a very broad sense has this study something to do with feedback; it is behavioristic reinforcement' (p9).
I checked the Standley (1996) and their results were interesting to me as I managed a band that played in pubs in Melbourne in the 1990's and the debate was about live music versus DJs or recorded music. But Standley (1996, p108) answered that question for us,
"Live music (ES = 1.13) was more effective than recorded music (ES = 0.86)."
Prof Terry Wrigley (2015) in Bullying by Numbers, critiquing the EEF in particular but also Hattie,
'Specifically, on Feedback, the Toolkit provides some more specific references to back up its very general claims, but many of these are over 20 years old and currently unobtainable. Seven more detailed references are given, each with an ‘effect size’, but these range from .97 to .20. Which is to be believed? Summaries follow, in highly technical language, mostly without indicating which stage or subject, what kind of learning, what kind of feedback, which countries the research took place in, and so on. Some of the sources are very critical of particular types of feedback... 
Meta-analyses are used in Medicine to enable researchers to complement the reading of other research, though not to substitute for it; for example, if experiments have been based on small samples, averaging the results can suggest a general trend. 
But the medical literature contains serious warnings against the misuse of meta-analysis. Statisticians are warned not to mix together different treatments, types of patient or outcome measures – the ‘apples and pears’ problem. If the original results differ strongly, they are advised to highlight the difference, not provide a misleading average. This is exactly what has not happened in the Toolkit, which should never have provided an average score for “Feedback” since the word has so many meanings' (p6).
Ruiz-Primo & Li (2013) detail the quality issues which we consistently see in Hattie's synthesis,
'A high percentage of papers investigating the impact of feedback did so without using a control group... 
Confounded effects, rarely mentioned in the synthesis and meta-analyses, pose another threat to validity when interpreting results of feedback studies' (p218).
'most of the studies do not provide information about the reliability and validity of the instruments used to measure the effects of feedback on the selected outcomes. The validity of feedback studies is threatened by a failure to attend to the technical characteristics of the instruments used to measure learning outcomes... Given these measures with ambiguity in technical soundness, can we fully trust results reported in synthesis and meta-analysis studies?
... there is an issue of ecological validity. For a research study to possess ecological validity and its results to be generalizable, the methods, materials, and setting of the study must sufficiently approximate the real-life situation that is under investigation. Most of the studies reported are laboratory-based or are conducted in classrooms but under artificial conditions (e.g., students were asked to identify unfamiliar uses of familiar objects)...
Furthermore, a high percentage of the studies focus on written feedback, and only a few on oral or other types of feedback, although oral feedback is more frequently observed in teachers’ daily assessment practices (see Hargreaves et al., 2000).
...we argue that formative feedback, when studied in the classroom context, is far more complex than it tends to appear in most studies, syntheses, or meta-analyses. Feedback practice is more than simply giving students feedback orally or in written form with externally or self-generated information and descriptive comments. We argue that feedback that is not used by students to move their learning forward is not formative feedback. We thus suggest that feedback needs to be examined more closely in the classroom setting, which should ultimately contribute to an expanded and more accurate and precise definition' (p219).
Some interesting findings,
'Research has made clear that students hardly read teachers’ written feedback or know how to interpret it (Cowie, 2005a, 2005b)' (p225).
'most of the publications on formative assessment and feedback include examples of strategies and techniques that teachers can use. Most of them, however, do not provide empirical evidence of the impact of these strategies on student learning; nor do they link them to contextual issues that may affect the effectiveness of the strategies...
there is a lack of studies conducted in real classrooms—the natural setting—where it would be important to see evidence that feedback strategies have substantive impact. Moreover, few studies have focused on feedback over extended periods or on factors that can moderate or mediate the effectiveness of feedback. Therefore, we cannot generalize what we know from the literature to classroom practices...
Rather than persisting with our common belief that feedback is something doable for teachers, we should strive to study formative assessment practices in the classroom, including feedback, to help teachers and students to do better. Given these unanswered questions, we need different and more trustworthy strategies of inquiry to acquire firsthand knowledge about feedback in the classroom context and to systematically study its effects on student learning' (p226).
David Didau in his excellent blog on feedback looks at the key studies used by Hattie and the EEF and confirms Wiliam's analysis and also shows feedback is complicated and nuanced.

He also pointed out that even though most researchers have feedback as an important teaching strategy, PISA has feedback NEGATIVELY correlated with Science performance.

The Negative Influences

From PISA 2015 Volume 2 (page 228).

Michael Pershan writes an excellent and insightful blog on feedback. He basic argument is the evidence is poor and the notion of feedback is too general to be of any help to teachers.

No comments:

Post a Comment