My claim
That the promotion of particular types of evidence has usurped Teacher Expertise and Agency & we need to challenge this.
Simpson (2019b) has written extensively on this,
"In the last 15 years or more, there has been a significant shift towards what has been called 'evidence-based education' (EBE). While the phrase may seem benign – after all, who would want to base practice on anything other than evidence.-proponents often appear to accept only particular meanings for the term 'evidence'."
Holmes et al. (2006) advise,
"Unmasking the hidden politics of evidence-based discourse is paramount..."
My position is consistent with the highly respected European academic Gert Biesta (2010), who summarises,
"...we have not yet conducted sufficient research in order to be able to encapsulate all factors, aspects and dimensions that make up the reality of education."
and also consistent with Dr Ben Goldacre on Meta-analysis in Education,
"I think you'll find it's a bit more complicated than that."
We need an open debate about the existing evidence claims and a focus on QUALITY research.
Prof Robert Slavin (2020) details the way forward-
Hattie has since retreated, in Hattie & Hamilton (2020),
Gert Biesta (2010) writes in detail warning of the demise of teacher agency as a result of the 'evidence-based' agenda,
Similar issues were discussed in the medical and health sciences, e.g., Holmes et al. (2006) warn,
In another report by Dean Ashenden, summarising and interviewing the then Education Minister, Christopher Pyne (2012), Ashenden states,
"The best single source of evidence about the relative effectiveness of class size reductions and many other educational strategies is John Hattie’s Visible Learning.."
Pyne was reported to say,
"the evidence overwhelmingly shows that investing in teacher effectiveness rather than the number of teachers is the most successful method of improving student learning and creating top performing education systems."
Professor Blatchford (2016c) comments about this,
Hattie attacks the easy target - The teacher:
Hattie summarises his book,
Everything she did had to be turned into a set of data to prove it had been carried out. “I was being constantly monitored and told off. I had no professional control at all.”
Whilst it may be simpler and easier to see teaching as a set of discrete influences, the evidence shows that these influences interact in ways in which no-one, as yet, can quantify. It is the combining of influences in a complex way that defines the 'art' of teaching.
"the value of a category of educational programs cannot be determined by its average effects on achievement. Rather, the value of the category should depend on the effectiveness of its best, replicated, and replicable examples."
Until that is the case we have to rely on Teacher Expertise.
The Deriding of Teacher Expertise
Hattie for over 10 years has spread his meme promoting his evidence:
Hattie continued with this type of polemic,
"statements without evidence are just opinions"This belittled Teacher Expertise & Experience, and raised his so-called evidence and rankings above them.
Hattie continued with this type of polemic,
"When teachers claim that they are having a positive effect on achievement or when a policy improves achievement, this is almost always a trivial claim: Virtually everything works. One only needs a pulse and we can improve achievement." (VL p. 16).Hattie's claim "Virtually everything works" is easily challenged by the studies Hattie cites, e.g., Hattie's prime Feedback study, Kluger & DeNisi (1996) report that 32% of feedback Does NOT work!.
Dylan Wiliam comments about this - see my page on Feedback.
Marzano (2009) also notes this and goes further,
"I’ve observed the same phenomenon with virtually every strategy and every innovation I’ve examined. I’ve come to the conclusion that you can expect anywhere from 20% to 40% of the studies in any given area to report negative results. While this might surprise or even discourage some, that’s simply the nature of the social sciences, in which there are a wide variety of variables that determine whether a particular strategy is going to produce positive results in any given situation. The lesson to be learned is that educators must always look to whether a particular strategy is producing the desired results as opposed to simply assuming that if a strategy is being used, positive results will ensue. If a strategy doesn’t appear to be working well, educators must adapt the strategy as needed or use other strategies. This is yet another reason why teachers shouldn’t be required to use specific strategies. Since none are guaranteed to work, teachers must have the freedom and flexibility to adapt or try something different when student learning isn’t forthcoming." (p. 34-35)
Sorta Work?
"Most things that a teacher could do in a classroom 'sorta' work…" (p. 3)
Nick Rose and Susanna Eriksson-Lee in their excellent paper 'Putting evidence to work', quote a more provocative slogan from Kevan Collins, Chief Executive of the Education Endowment Foundation (EEF) & and recently appointed Board member of the Australian Education Research Organisation (AERO),
"if you're not using evidence to inform your decisions, you must be using prejudice."Why Has Hattie Become so Popular?
Scott Eacott (2017) in School Leadership and the cult of the guru: the neo-Taylorism of Hattie, provides a simple answer,
"Hattie’s work has provided school leaders with data that APPEAL to their administrative pursuits."
A Warning
Gert Biesta (2010) writes in detail warning of the demise of teacher agency as a result of the 'evidence-based' agenda,
"my contribution is primarily meant to provide educators and other professionals with arguments that can help them to resist unwarranted expectations about the role of evidence in their practices and even more so of unwarranted interventions in their practices" (p. 491).
"This becomes deeply problematic in those cases in which it is argued that professionals should only be allowed to do those things for which there is positive research evidence available - an approach which Holmes et al. (2006) have, in my view, correctly identified as a form of totalitarianism...
I have particularly highlighted the ‘democratic deficit’ of the uptake of the idea of evidence-based practice in education, emphasising how a particular use of evidence threatens to replace professional judgement and the wider democratic deliberation about the aims and ends and the conduct of education" (p. 492).
"Calling the idea of value-based education an alternative, is not meant to suggest that evidence plays no role at all in value-based education but is to highlight that its role is subordinate to the values that constitute practices as educational practices...
It also suggests that rather than there being a mechanistic connection between evidence and truth, there is a need for judgement about the relative weight of what is being submitted as evidence for a particular belief or proposition" (p. 493).
"...we have not yet conducted sufficient research in order to be able to encapsulate all factors, aspects and dimensions that make up the reality of education" (p. 494).
"...the so called ‘knowledge-base’ for practice is never sufficient and never will be sufficient. This, in turn raises the question how this gap is closed in practice - something to which I will return below" (p. 496).
"The teleological character of education provides us with one important reason for suggesting that questions about ‘what works’ - that is questions about the effectiveness of educational actions - are always secondary to questions of purpose" (p. 500).Then in 2015, Biesta argues,
"that three tendencies that are often presented as developments in the ongoing professionalisation of teaching and that can be found in different forms and guises in schools, colleges and universities - treating students as customers; being accountable; and replacing subjective judgement with scientific evidence - are undermining rather than enhancing opportunities for teacher professionalism. Taken together, the two lines of the article provide indications as to how teacher professionalism might be regained and reclaimed in the context of the discussion about education and its purpose" (p. 75).
"We must also bear in mind that the word ‘learning’ can refer to a very wide range of phenomena. Think, for example, of the difference between what it means to learn to ride a bike, to learn the second law of thermodynamics, to learn to be patient, to learn that you are not good at some-thing, etc.This is another reason why the suggestion that education is simply about making students learn or about facilitating their learning is potentially misleading, both for students and for teachers" (p. 76).
"the question of purpose is the most fundamental one for the simple reason that if we do not know what it is we are seeking to achieve with our educational arrangements and endeavours, we cannot make any decisions about the content that is most appropriate and the kind of relationships that are most conducive...
the question of purpose is a multidimensional question because education tends to function in relation to a number of domains... I have suggested that three domains can be found, viz., qualification, socialisation and subjectification" (p. 77).
"This is what we are witnessing with the current emphasis on achievement in the domain of qualification where excessive pressure on students (and teachers, for that matter) to perform in that domain (and within that domain in a very small number of subjects) is beginning to have a negative impact in the domain of subjectification. To put it bluntly: excessive emphasis on academic achievement causes severe stress for young people, particularly in cultures where failure is not really an option" (p. 78).
"But one-sidedness always comes at a price, the price we are willing to pay for a temporary emphasis on one of the dimensions. I wish to highlight once more that the current emphasis in many countries and settings on just enhancing academic achievement - i.e. performance in the domain of qualification - comes at a very high and potentially too high price" (p. 79).
"I am surprised by Hattie’s suggestion - partly made in response to my critique of evidence-based education (Biesta, 2007) - that, although there is more to education than academic achievement, in the end this is what is supposed to matter most (Hattie, 2008, pp. 245–255), thus reinforcing a one-dimensional view of education in which only qualification seems to count" (p. 80).
"If education requires judgement, and if this judgement is ‘of the teacher,’ then it would follow that teachers have ample space and opportunity to exercise such judgement. Yet it is here that we encounter problems in the ways in which the professional space for teachers is currently being constructed and ‘policed’ - ways that often limit rather than enhance the scope for teacher professional judgement" (p. 81).
"it would appear that we should welcome and embrace recent developments that emphasise the importance of seeing patients or students as customers who need to be served and satisfied; of taking the operation of the professions entirely transparent so that they can be even more accountable; and of basing professional activity on scientific evidence about ‘what works’ rather than on subjective judgements of individual professionals. While at first sight this may sound plausible and desirable - and demands for a focus on the customer, for transparency and for evidence-based ways of working are often ‘sold’ in this way - I would like to suggest that these developments may risk doing the opposite of what they claim to do and hence may lead to the erosion of responsible, accountable and democratic professionalism" (p. 82).
"While accountability in itself is a good and important idea - professionals need to be accountable both to the immediate clientele they serve and to the wider public - there is a crucial difference between democratic forms of accountability that engage in substantive exchanges between professionals and their ‘stakeholders’ about what, in the case of teaching, good education is and what the parameters for identifying good education are, and the bureaucratic forms of accountability that greatly ‘trouble’ contemporary education (Sahlberg, 2010). If democratic accountability focuses on what makes education good, bureaucratic accountability has transformed the practice of providing data in order to show how education meets certain pre-defined standards into an aim in itself, where questions about whether the standards that are being applied are accurate and meaningful expressions of what good education is supposed to be are no longer at the centre of the process...
while in theory ‘the new culture of accountability and audit makes professionals and institutions more accountable to the public’, in practice ‘the real requirements are for accountability to regulators, to departments of government, to funders, to legal standards’ (O’Neill, 2002)... while again in theory ‘the new culture of accountability and audit makes professionals and institutions more accountable for good performance’, in practice ‘the real focus is on performance indicators chosen for ease of measurement and control rather than because they measure accurately what the quality of performance is’ (O’Neill, 2002). The predicament here is whether we are measuring and assessing what we consider valuable, or whether bureaucratic accountability systems have created a situation in which we are valuing what is being measured, i.e. a situation where measurement has become an end in itself rather than a means to achieve good education in the fullest and broadest sense of the term" (p. 83).This importance of this issue has warranted a Special Edition - Vol 25 of Educational Research & Evaluation. The key academics, Adrian Simpson, Dylan Wiliam & Terry Wrigley also question the 'evidence-based' agenda and the dominance of Hattie and the EEF.
Similar issues were discussed in the medical and health sciences, e.g., Holmes et al. (2006) warn,
"Unmasking the hidden politics of evidence-based discourse is paramount..." (p. 181).
"A starting point for health sciences would be to promote the multiplicity of ... subjugated forms of knowledge... these forms of knowledge are ways of understanding the world that are 'disqualified as non nonceptual knowledges, as insufficiently elaborated knowledges: naïve knowledges, hierarchically inferior knowledges, [and] knowledges that are below the required level of erudition or scientificity’. These forms of knowledge arise from below, as it were, in contradistinction to the top-down approach that characterises the hegemonic thrust of EBHS [evidence based Health sciences]...
In our view, this positive process begins with a critique of EBHS and its hegemonic norms. As we have argued, according to postmodern authors, these norms institute a hidden political agenda through the very language and technologies deployed in the name of ‘truth’" (p. 183).
"This new guide book of scientific vocabulary, including terms connected with EBM (e.g. systematic literature review, knowledge transfer, best practices, champions, etc.), is taken seriously in the realm of health sciences, so much so that it is considered vital as a reflection of ‘real science’. The classification of scientific evidence as proposed by the Cochrane Group thus constitutes not only a powerful mechanism of exclusion for some types of knowledge, it also acts as an organising structure for knowledge and a mechanism of ideological reinforcement for the dominant scientific paradigm. In that sense, it obeys a fascist logic" (p. 184).
Teacher Effects?
Based on his 2003 presentation, Teachers Make a Difference, What is the research evidence? Hattie claimed that teachers account for 30% of the effect on student achievement.
"There have been many studies over the past few years that have asked this question about wherein lies the variance. Most have been conducted using Hierarchical Linear Modelling..."
But Hattie gives no specific reference to check this 30% figure.
He displays this graphic in most of his presentations:
Yet, Pasi Sahlberg (2015) You can do more with less, makes a totally different claim:
"Research on what explains students’ measured performance in school remains mixed. However, researchers generally agree that up to two-thirds of the variation in student achievement is explainable by individual student characteristics like family background and such variables. The American Statistical Association concluded recently that teachers account for about 1-14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in system-level conditions. In other words, most of what explains student achievement is beyond the control of teachers or even schools, and therefore arguing that teachers are the most important factor in improving the quality of education is simply wrong.
This doesn’t mean that teachers would not be important or that individual teachers could not turn the course of children in school. Of course they do. But it is often a combination of powerful factors that makes the most positive impact on students. Most scholars agree that effective leadership is among the most important characteristics of good schools, equally important to powerful teaching.[Note: Hattie has a low effect size of 0.36 for leadership]. Effective leadership includes leader qualities, such as being firm and purposeful, having a shared vision and goals, promoting teamwork and collegiality and frequent personal monitoring and feedback. Several other characteristics of more effective schools include features that are also linked to the culture of the school and leadership: maintaining focus on learning, producing a positive school climate, setting high expectations for all, developing staff skills and involving parents. In other words, school leadership matters as much as teacher quality."
Professor Richard Haier also reports that the Coleman Report deduces that Teacher + School effects amount to about 10% of Student Achievement, the rest is what the student brings to school. He also cites the Gates Foundation funding and their focus on teacher effects and then pulling away from this work when not much improvement was measured - see here.
Grasby et al. (2020) in a well-designed twin study discuss Hattie's claim of 30%, report their findings of 2-3% and conclude that their,
"results are at odds with claims in public discourse of substantial classroom-level influences, which are mostly portrayed as teacher effects."
Another report from McKinsey & Company (2022) showed teacher effects from 15-19% and Mindset being a more important factor-
Similarly, the Shanker Institute estimates teacher effects to be 10-15% - details here.
Also, in a useful discussion on Teachers Talk Radio, Robert Plomin, Dylan Wiliam & Stephen Gorard all agree that teacher effects are small compared to genetic and family factors. Gorard's comments @ 38mins are very interesting.
The Politics?
Australian Education Minister Chris Pyne used Hattie's 30% claim to direct Policy.
However, Professor Alan Reid (2014), Decisions about teaching methods should be made by educators not politicians, questions Pyne's starting assumption from Hattie that, ‘research demonstrates that the quality of the teacher is the most important in-school factor which promotes student learning’.
Reid states,
"Once this is done it is possible to claim that all the money spent on these peripheral matters has resulted in reduced learning outcomes, and hey-presto, you have an educational justification for reducing expenditure. It is not surprising that Minister Pyne finds this an attractive thesis.The problem is that it is nonsense. It is the interrelationship of the variables in the context of the learning which is important. They cannot sensibly be separated out in this way.But having isolated teacher quality, the government is able to focus on those strategies which it claims will enhance it. This demands a view of what good teaching looks like – something Minister Pyne has not been shy to articulate."
Reid then quotes Pyne's notion of good teaching - Direct Instruction.
"The best single source of evidence about the relative effectiveness of class size reductions and many other educational strategies is John Hattie’s Visible Learning.."
Pyne was reported to say,
"the evidence overwhelmingly shows that investing in teacher effectiveness rather than the number of teachers is the most successful method of improving student learning and creating top performing education systems."
Professor Blatchford (2016c) comments about this,
"When Christopher Pyne [the then Australian Education Minister] talked about prioritising teacher quality, rather than reducing class sizes, he set up a false and simplistic dichotomy" (p. 16).
Then in New Zealand,
Professor John O'Neill has, for a long time, warned of the effect on policy by Hattie. In, Material fallacies of education research evidence and public policy advice, O'Neill (2012b) states,
"...the Minister of Education declined to rule out increases in class size. In short, this was because the ‘independent observation’ of Treasury and the research findings of an influential government adviser, Professor John Hattie, were that schooling policy should instead focus on improving the quality of teaching."
Prof Larsen (2014) Know thy impact – blind spots in John Hattie’s evidence credo, takes this further,
"Teachers get identified as the primary and indispensable learning factor and thereby as a public, expensive, and untrustworthy potential enemy. This amounts to scapegoat projection par excellence..." (p. 11).
Hattie's claim is that students bring a whole host of personal and social factors with them that School's can't change. Also, systemic aspects, class-size, buildings, uniform, summer school, finance, etc., don't effect student achievement much. So the teacher is what we should focus on.
In his excellent analysis, School Leadership and the cult of the guru: the neo-Taylorism of Hattie, Professor Scott Eacott says,
"Hattie’s work has provided school leaders with data that appeal to their administrative pursuits" (p. 3).
"The uncritical acceptance of his work as the definitive word on what works in schooling, particularly by large professional associations such as ACEL, is highly problematic" (p. 11).
Prof Robyn Ewing on Insight (SBS TV),
"As teachers, we continue to be blamed for all of the things that society doesn't get right."
Hattie summarises his book,
"the devil in this story is not the negative, criminal, and incompetent teacher, but the average, let's get through the curricula… teacher" (p. 258).
This is an amazing critique and represents Hattie's focus throughout the book. He seems oblivious to systemic and political influences and seems all too eager to focus the blame on the easy target - the teacher.
Professor Gunn Imsen is also troubled by Hattie's claim,
Yet Hattie says,
Professor Gunn Imsen is also troubled by Hattie's claim,
"These are strong and sensational words."In Hattie's jurisdiction, Victoria, Australia; teachers can get dismissed for not teaching the state defined curricula, particularly Year 11&12 - click here for examples.
Yet Hattie says,
"Educating is more than teaching people to think – it is also teaching people things that are worth learning" (p. 27).
This is the realm of politicians and senior bureaucrats, who mostly decide what is worth learning by designing and enforcing a curriculum.
So if following the curricula is the issue, why not focus on those who decide the curriculum? They are not the teachers!
So if following the curricula is the issue, why not focus on those who decide the curriculum? They are not the teachers!
Visible Learning: the Sequel - Hattie's 2023 update
Hattie removes his claim that teacher's account for 30% of the variation in student achievement and mentions the wide variation in the results other studies have found, e.g., Grasby et al. (2020) - low result of 2-3%.
However, Hattie still claims,
"The teacher accounts for the greatest source of within-school variance." (p, 216)
A Teacher's Lament:
Gabbie Stroud resigned from her teaching position and wrote:
Stephanie Wescott gave a teacher's perspective in the Sydney Morning Herald,Gabbie Stroud resigned from her teaching position and wrote:
"Teaching – good teaching - is both a science and an art. Yet in Australia today [it]… is considered something purely technical and methodical that can be rationalised and weighed.
But quality teaching isn't borne of tiered 'professional standards'. It cannot be reduced to a formula or discrete parts. It cannot be compartmentalised into boxes and 'checked off'. Good teaching comes from professionals who are valued. It comes from teachers who know their students, who build relationships, who meet learners at their point of need and who recognise that there's nothing standard about the journey of learning. We cannot forget the art of teaching – without it, schools become factories, students become products and teachers: nothing more than machinery."Dylan Wiliam, after watching a Bill Moyers (A World of Ideas, 1989) interview with Sarah Lawrence Lightfoot who defined effective teaching as ideas conveyed through relationships (p. 159); says,
"probably the best 4-word definition I have seen."
"However, the greatest undermining of the profession comes from the bureaucratic initiatives that claim to work to improve teacher quality. These include the requirements that teachers produce and analyse data to inform decision-making, enforcing school-wide templates for lesson-planning, and stringent performance review mechanisms, as well as a host of administrative requirements. These work to remove teacher agency, demand greater reporting and accountability and suggest - of most concern - that teachers cannot be trusted to work independently.
While there is nothing inherently wrong with the rigour of evidence-informed practice models, or in requiring teachers to work with data in their practice, these trends do reinforce the notion that teacher expertise exists outside our own professional practice.
The increasingly common requirement that teachers look to standardised models of practice, where strategies are selected from a narrow evidence-base, suggests that distrust for teachers as qualified professionals is increasing. These trends also fail to acknowledge the incredible unpredictability of working with young people; what works in one place may not replicate in another."Emma Kell, a secondary school teacher and author of How to Survive in Teaching, said she had felt constantly stressed while working a 70-hour week – and not just because of the workload. “It was the feeling of never being trusted, and my voice not being heard,” she said, adding that Education Support had helped her to survive.
Everything she did had to be turned into a set of data to prove it had been carried out. “I was being constantly monitored and told off. I had no professional control at all.”
Whilst it may be simpler and easier to see teaching as a set of discrete influences, the evidence shows that these influences interact in ways in which no-one, as yet, can quantify. It is the combining of influences in a complex way that defines the 'art' of teaching.
Is It Burnout? Or Demoralization?
Doris Santoro analyses the high rates of teacher burnout and changes this to demoralization.
Read full analyses here.
For Australian Education Union members - A Motion to put forward
The Trust and Belief in Experience comparisons are illustrative of the difference between leadership and teachers.
https://teachingbattleground.wordpress.com/2019/09/28/teacher-autonomy-is-the-most-difficult-issue-in-the-education-debate-part-1/
Books dealing with Teacher Agency:
Flip the System Australia and Jelmer EVERS.
Prof Alan Reid on the demise of Teacher Agency - summary here.
Don't blame the teacher: student results are (mostly) out of their hands.
New South Wales Teachers Federation Inquiry 2020 - Valuing the Teaching Profession.
Guide to Management Styles - here.
We the Sub Branch at ___________________ School thank the AEU executive for starting a campaign to improve Teacher Agency by creating a Curriculum Policy document designed by teachers.
We also acknowledge that aspects of the “evidence based agenda” have been addressed in the AEU’s Professional Voice Journal. However, given the dominance of the Meta-meta-analysis technique of Hattie, in Australian Educational Policy, we also ask the AEU executive scrutinise that evidence and provide a statement to members outlining:
1. The extensive critique of this technique as a valid method for determining "high impact" or "high yield" teaching strategies amongst Hattie's education researcher peers.
2. A challenge to Hattie's attack on class size was produced by the AEU, using class size research experts Profs Peter Blatchford & David Zyngier, but it was not promoted. We ask for further promotion of their detailed work. Also, we ask for further challenge by looking in detail at the 3 - meta-analyses Hattie used. We believe an examination of these will show the extent to Hattie's misrepresentation more strongly.
3. The validity of evidence derived from other types of research, e.g. case & longitudinal study, etc.
4. The validity of evidence generated by teachers through their professional practice.
5. The financial conflicts of interest of the players promoting the Meta-meta-analysis evidence base (as the New Zealand Education Unions have done).
6. The promotion of the peer review critique through the AEU’s Teacher Learning Network.
7. Professional Development of all council members.
Teacher Attitudes
The Monash University Q Project has done extensive surveys regarding Teacher & Principal attitudes to evidence, e.g., "Educators Research Use Awareness & Attitudes Survey",
The Trust and Belief in Experience comparisons are illustrative of the difference between leadership and teachers.
Teachers generally don't trust research and believe more in their experience.
More examples of Teachers responses in Q survey:
The Australian Curriculum Review
Maths researcher Marty Ross in his series of posts, summarises well, starting with the title,
A detailed analysis of Teacher Autonomy:
https://teachingbattleground.wordpress.com/2019/09/28/teacher-autonomy-is-the-most-difficult-issue-in-the-education-debate-part-1/
Toxic executives in schools
The Ripple effect of toxic executives in schools, an analysis - here.
The authors identified three factors that permit destructive leadership:
The hierarchical organisation of schools and the prevailing power relations in favour of those in authority.
The personality dispositions of some leaders, exhibiting characteristics associated with negative traits such as narcissism, psychopathy or Machiavellianism.
The nature of relationships between leaders and followers and the sociological and psychological susceptibility of some subordinates to the behaviours and actions of the leader.
Books dealing with Teacher Agency:
Flip the System Australia and Jelmer EVERS.
Don't blame the teacher: student results are (mostly) out of their hands.
New South Wales Teachers Federation Inquiry 2020 - Valuing the Teaching Profession.
Guide to Management Styles - here.
thanks James
ReplyDelete