Other Researchers

Why do key researchers differ significantly & even contradict each other when their top strategies are compared?

For example, Daniel Willingham, a key researcher and advisor to the Evidence Organisation Deans for Impact in his influential book "Why Don't Students Like School", totally contradicts Hattie's claim in VL that it is possible to meaningfully separate & compare individual educational influences (VL, p. 4) - video here.

Willingham says there is "a big gap between research and practice" and influences "cannot be separated in the classroom" as "they often interact in difficult-to-predict ways." He provides the following example, 
"... laboratory studies show that repetition helps learning, but any teacher knows that you can’t take that finding and pop it into a classroom by, for example, having students repeat long-division problems until they’ve mastered the process. 
Repetition is good for learning but terrible for motivation. With too much repetition, motivation plummets, students stop trying, and no learning takes place. The classroom application would not duplicate the laboratory result." (from the introduction).
Willingham also in his book, When Can You Trust the Experts?, details these warning signs about educational research,

Is the theory related to a single individual or Guru?

Do the findings contradict the collective experience of teachers?

Is the context of the evidence sufficiently similar to ours?

Is the research peer reviewed?

Hattie's theory fails on ALL of these.


1. Hattie, Marzano, and Dylan Wiliam:


Hattie's Top Strategies 2009Hattie's Top Strategies 2017MarzanoProf Dylan Wiliam
1. Self-Reported GradesCollective Teacher EfficacyIdentify Similarities+DiffsFormative Assessment
2. Piagetian programsSelf-Reported GradesSumm + Note-takingFeedback
3. Formative evaluationTeacher estimates of achReinforcing Effort+RecognitionPeer Tutoring
4. Micro-teachingCognitive task analysisHomework and PracticeSelf-regulated learning
5. AccelerationResponse to interventionNonlinguistic RepresentationsDeliberate practice
6. Classroom behavioralPiagetian programsCooperative LearningTeacher Mindset
7. Intervent-learning disabledJigsaw methodSetting Objectives+FeedbackStaff as critical friends
8. Teacher clarityConceptual changeGenerating and Testing Hypoth's
9. Reciprocal teachingPrior AbilityCues+Advanced Orgs
10. FeedbackIntegrate prior knowledge

Hattie now admits his rankings are misleading! Listen to Ollie Lovell interview with Hattie in June 2018 (1hr 21min 45sec) here.

'it worked then it got misleading so I stopped it'!!!

Marzano has been criticised by Dr J BeckerEllis Whyatt and Matthew Fitzpatrick for similar reasons to Hattie - misrepresentation, poor research and conflicts of interest.

2. The What Works Clearinghouse (WWC)

The U.S Education Department fund a research organisation called the What Works Clearinghouse (WWC) consisting of 20+ distinguished professors & PhD research scientists. 

They review research with a focus on the quality of research and have a stringent quality control standards, e.g., 



As already stated, most of Hattie's studies would fail the WWC standards.

It is significant that the WWC provide some caution in their recommendations with the 'Level of Evidence' rating. This is at odds with Hattie's confidence about his own conclusions. Also, one of the strong features of these practice guides is that they are subjected to rigorous external peer review. Hattie rarely responds to the critique from numerous peer reviews of his work.

WWC Recommendations:




U.S. Ed Dept Recommendation versus HattieLevel of EvidenceHattie's Rank
Space learning over time.Moderate70?
Interleave worked example with problem-solving exercises.Moderate20,30,118?
Combine graphics with verbal descriptions.Moderate104
Connect and integrate abstract and concrete representations.Moderate82?
Use pre-questions to introduce a new topic.Minimalnot ranked
Use quizzes to re-expose students to key content.Strongnot ranked
use delayed judgments of learning to identify.Minimalnot ranked
Tests and quizzes to identify content that needs to be learned.Minimal3?
Ask deep explanatory questions.Strong53?

A colleague of mine, Oliver Lovell, has used strategies closely related to the U.S recommendations to improve his Year 12 Maths class score by 1 standard deviation; quite an improvement! Details of what Ollie did here.

The U.S Dept of Education Maths specific Recommendations:



3. The Finnish System:

The Director-General Pasi Sahlberg outlines in Finnish Lessons 2.0, the priorities of the Finnish system, Teacher Training both in subject knowledge and didactics (p. 77).

Mathematics teaching is strongly embedded in curriculum design and teacher education in Finnish primary schools (p. 77).

A focus on welfare policies, e.g free healthy lunch for all children (p. 62).

Student's have choice, 
'Today, students build their own personalized learning schedules from a menu of courses offered in their school or by other education institutions. Studying in upper-secondary school is therefore flexible, and selected courses can be completed at a different pace depending on students’ abilities and life situations' (p. 87).
Systematic counselling and career guidance (p. 87).

School Autonomy,
'little interference by the central education administration in schools’ everyday lives' (p. 88).
Less instructional time and more time for play and recreation (p. 91).

Less teacher workload,
'Lower-secondary teachers’ total weekly working time in Finland was 31.6 hours; that is significantly less than in Australia (42.7 hr), the United States (44.8), England (45.9), Singapore (47.6), Alberta (48.2), or in the surveyed 34 countries on average (38.3)' (p. 91).
Time for staff collaboration,
'... teaching is a holistic profession that combines work with students in the classroom and collaboration with colleagues in the staff room' (p. 93).
Teacher-designed curricula (p. 99). 

Systematic care for students with diverse special needs (p. 99).

4. The Education Endowment Foundation (EEF):

Is an independent English organisation with a funding of around $250 Million. It has 5 full-time researchers and has published an evidence guide mostly based on good quality Randomised Controlled Studies (RCTs) similar to the US Education Dept, but adds a helpful costing feature.


It has a very different time scale to Hattie. An effect size of about d = 0.1 equivalent to a months progress. In contrast, Hattie has an effect size of 0.4 equivalent to 12 months progress.

Their ranking of influences is very different to Hattie's - click here for full list:




5. The Sutton Trust: 

With a focus on good quality research, the two factors with the strongest evidence of improving pupil attainment are:

a. Teachers’ content knowledge, including their ability to understand how students think about a subject and identify common misconceptions - a direct contradiction to Hattie's work where this is ranked #125 with a small effect size = 0.09.

b. The quality of instruction, which includes using strategies like effective questioning and the use of assessment.

Full Paper - What Makes Great Teaching here.

6. The Deans For Impact (20+ Deans of Educational faculties): Intro Video.





They produced a document The Science of Learning which details the key strategies for effective teaching, e.g.,

a. Students learn new ideas by reference to ideas they already know.

b. Information is often withdrawn from memory just as it went in. We usually want students to remember what information means and why it is important, so they should think about meaning when they encounter to-be-remembered material.

c. Practice is essential to learning new facts, but not all practice is equivalent.

d. Self-determined motivation (a consequence of values or pure interest) leads to better long-term outcomes than controlled motivation (a consequence of reward/punishment or perceptions of self-worth).

e. Effective feedback is often essential to acquiring new knowledge and skills.

f. Students will be more motivated and successful in academic environments when they believe that they belong and are accepted in those environments.


The Deans for Impact just released (March 2019) their Science of Early Learning.

7. Barak Rosenshine's Principals of instruction:

Rosenshine's 2012 paper - here.

Oliver Caviglioli’s summary poster.

A number of teachers cite that these principles are the one that should be used by teachers, e.g., Nick Rose,

"One of the problems with the latest research is that the conclusions are necessarily tentative and there’s a good chance that the next researcher might identify something that contradicts it. This leaves teachers with a problem when trying to identify evidence-informed approaches to developing their teaching: is it worth embarking on something involving lots of time and effort, only to discover that researchers change their minds in a year’s time?

One way around this problem is to look for findings that have been triangulated. For example, if we find an outcome in well-controlled, but quite artificial, laboratory experiments and we find the same result in authentic classroom settings, despite all the noisy variables involved, then that likely makes it a good bet to try to implement it in your classroom.

In this paper, Barak Rosenshine reviews different bodies of research, including cognitive science and classroom studies, to identify where the science and practice appear to tell us the same thing about how we might take a research-informed approach to improving our teaching. If you have time to read only one research summary this year, I would recommend this one."


8. Professor Jo Boaler:

Jo Boaler's focus is on visual representations of abstract ideas which seems to directly contradict Hattie's rankings where 'simulations' and 'visual representations' are ranked very lowly.



And Jo Boaler's example of simulations:




Although Boaler's use of the mindset research has been criticised see here.

9. Professor Michael Fullan:

In ACEL Monograph 52 Fullan states,
'there was one finding that stood out as twice as powerful as any other factor in 'effect size' - principals who participated as learners working with teachers to make improvements had twice the impact on school-wide student achievement compared to any other factor' (p. 3).
Note that Fullan is a leadership consultant with a focus on Leadership.

10. The Grattan Institute:

Professor Wiliam also provides some extra useful strategies on the teacher. He suggests teachers must want to improve and teachers need to act as critical friends. The Grattan Institute analysed the high performing international educational systems and concluded that one of the reforms responsible for improving student achievement across the four high-performing education systems in East Asia was teachers acting as critical friends. Note the amount of time devoted to feedback, lesson planning, etc. Although, one caveat in these systems is that class sizes are higher (Grattan Report p. 14).




















11. PISA 2015:

David Didau in his excellent blog on feedback pointed out that even though most researchers have feedback as an important teaching strategy, PISA has feedback NEGATIVELY correlated with Science performance.



The Negative Influences


From PISA 2015 Volume 2 (p. 228)

12. Others:

The Institute for Effective Education.

Prof Robert Slavin - The Fabulous 20%: Programs Proven Effective in Rigorous Research.

Center for Research and Reform in Education at Johns Hopkins University - Proven Programs. Their quality evidence levels:

Strong evidence: At least one well-designed and well-implemented experimental (i.e., randomized) study.

Moderate evidence: At least one well-designed and well-implemented quasi-experimental (i.e., matched) study.

Promising evidence: At least one well-designed and well-implemented correlational study with statistical controls for selection bias.

13. Student Research on What makes a great teacher:

Azul Terronez surveyed 26,000 students and found great teachers:

1. build positive relationships.
2. are chilled.
3. are good listeners.
4. love to learn.
5. knows kids have a life outside of school.
6. notice if kids struggle.
7. sings!
8. are humble and take risks.

His TEDx talk:



The Australian Government Productivity Commission.

In their latest report (2017) they make a number of recommendations regarding educational research.

Firstly, a focus on the quality of research:

'... the gold standard techniques are meta-analyses of randomised controlled trials and individual trials. Such approaches are widely used in health research, but are not routinely used in Australian education research' (p. 17).

Verifying the quality of the research (p. 19):

'A range of processes can be used to ensure the findings from completed research are robust. These include independent validation of the findings, peer review of research, publication of all outputs to enable scrutiny and debate (irrespective of findings), and the provision of project data for secondary analysis.'

Their recommendations (p. 28):

In assessing whether to improve the quality of existing education data, governments should examine on a case by case basis whether:
the existing quality of the data is fit for purpose.
data quality improvements are feasible given the context of data collection.
other options are available.
the benefits of improving data quality exceed the costs.

The Australian Government should request and sufficiently fund the agencies that conduct the Longitudinal Study of Australian Children to establish new cohorts at regular intervals. 


U.S. Ed Dept Levels of Evidence guide:


The US Education department state, 'In classifying levels of empirical support for the effectiveness of our recommendations, we have been mindful not only to the issue of whether a study meets the “gold-standard” of a randomized trial but also to the question “Effective as compared to what?” Virtually any educational manipulation that involves exposing students to subject content, regardless of how this exposure is provided, is likely to provide some benefit when compared against no exposure at all. To recommend it, however, the question becomes “Is it more effective than the alternative it would likely replace?” In laboratory studies, the nature of instruction in the control group is usually quite well defined, but in classroom studies, it is often much less clear. In assessing classroom studies, we have placed the most value on studies that involve a baseline that seems reasonably likely to approximate what might be the ordinary practice default' (p. 3).


Full Guide here https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf































No comments:

Post a Comment