Collective Teacher Efficacy

Collective Teacher Efficacy (CTE) has Hattie's highest effect size of d =1.57.

Hattie's Claim

Hattie & Hamilton (2020) in Real Gold vs Fool's Gold stated, 
"From the research, we know that the following things matter the most: 
1. Achieving teacher collective efficacy.
...The research tells us that where teachers have this shared belief, great things happen." (p. 22)
Yet, they cautioned, 
"...a key point of caution is that there is currently only one meta-analysis of 26 research studies. The evidence base is still too small to form anything more than speculative conclusions." (p. 19)
Hattie's new rating - OVERALL CONFIDENCE, rates CTE very LOWLY as 2 out of 5.

So, from these contradictory ratings, it is difficult to know if CTE is real gold or fool's gold!

These contradictions are discussed with Hattie's new mantra - The Story, the Story, the Story.

The Evidence

Hattie originally used one meta-analysis to determine his effect size, a PhD thesis:

Eells, Rachel Jean. (2011) Meta-Analysis of the Relationship Between Collective Teacher Efficacy (CTE) and Student Achievement. Loyola University of Chicago.

However, a 2nd meta-analysis has recently been added, Çoğaltay & Karadağ (2017), which reports a correlation r = 0.52 (p. 221). Hattie then converts this to an effect size (d) = 1.22. 

Note: Hattie's conversion of a correlation to an effect size, then comparing this with different methods of effect size calculation has been widely critiqued in the peer review - see Correlation.

Çoğaltay & Karadağ (2017) used 36 studies, while Eells (2011) used 25 studies. 

Hattie's method of averaging these TWO meta-analyses to get ONE effect size has also come under significant peer review critique (Shannahan, 2017 & Wecker et al., 2017) as meta-analyses often include the SAME studies. So Hattie's averaging counts the same study twice, or even multiple times. This causes significant bias to the effect size calculation - see Effect Size.

In this regard, Çoğaltay & Karadağ (2017) used the same 21 out of 25 studies that Eells (2011) used.

Removing duplicate studies is a major protocol of the meta- analysis method, as shown by Eells below. But, Hattie had ignored this and many other protocols for over 20 years.

However, due to the weight of peer review critique, Hattie has finally admitted this is a major problem in his work and has re-examined his feedback research, Wisniewski, Zierer & Hattie (2020) "The Power of Feedback Revisited", getting totally different results,
"...a source of distortion when using a synthesis approach results from overlapping samples of studies. By integrating a number of meta-analyses dealing with effects of feedback interventions without checking every single primary study, there is a high probability that the samples of primary studies integrated in these meta-analyses are not independent of each other...Therefore, these would have to be considered as duplets–primary studies that are included in the result of the synthesis more than once–and consequently cause a distortion." (p. 2)
Hattie is a master wordsmith, while he used the soft word 'distortion' other peer reviews use the stronger word BIAS

Also, it is strange that in 2020, Hattie admits there is a problem with duplicate studies in his representation of Feedback, but at the same time, adds the meta-analysis of Çoğaltay & Karadağ, which used 21 out of 25 of the same studies as Elle's used!

In addition, given this precedent, Hattie now must re-examine ALL of his other (more than 200) influences including CTE!

Hattie's use of the two meta-analyses also illustrate the unreliability of his method, as Çoğaltay & Karadağ (2017) is peer reviewed and includes most of the studies used by Eells (2011) PhD, therefore is a higher quality study. Using the effect size from Çoğaltay & Karadağ alone, (d) = 1.22 dropping CTE from #1 down to #5.

Other peer reviews also discuss the unreliability of Hattie's averaging.

Pant (2014) shows how influences can easily fall below Hattie's magic effect size, d = 0.40 (Zone of Desired effects) just by the method of averaging.

Wecker et al. (2017) gives many examples of Hattie's "flawed" calculations,
"This would mean a descent from 26th place to 98th in his ranking." (p. 31)
See (2017) commenting on the method of Hattie, Marzano & EEF,
"Averaging effect sizes from across studies of different quality giving equal weights to all can lead to misleading conclusions." (p. 10)

Comparison with Other Evidence Organisations

A key question for teachers is, why are the evidence summaries from different organisations so different and contradictory?

For example, the dominant U.K organisation, The Education Endowment Foundation (EEF), uses the same method as Hattie, the Meta-meta analysis. Yet, CTE does not appear in any of their recommendations.

Another example is the largest education evidence organisation, the USA, What Works Clearing House (WWC). It focuses on the highest quality of evidence - the individual study, based on random allocation of students. Once again, CTE does not appear in any of their recommendations.

Another example is the large English organisation Evidence Based Education. In their Great Teaching Toolkit, they don't mention CTE either and they warn about these types of correlation studies,

"Much of the available research is based around correlational studies; in these the relationships between two variables is measured. While interesting, the conclusions drawn from them are limited. We cannot tell if the two have a causal relationship – does X cause Y, or does Y cause X? Or might there be a third variable, Z? Therefore, while we may find a positive correlation between a teaching practice and student outcomes, we do not know if the practice caused the outcome." (p. 11)

The Critical Peer Review

Hoogsteen (2020) comments,
"Arguing against the existence or the positive effects and outcomes of CTE was never the intent of this article, nonetheless, examination and critique of the quantitative studies involving CTE can further support the contention that the field of education needs to reconsider the status currently ascribed to the concept." (p. 5)
Also, Hoogsteen (2021) suggests a wise Action Research of CTE implementation and concludes that it may be time to cease viewing collective efficacy as a shaper of the normative environment of a school but rather a characteristic to build. (p. 82)

Zhao (2019) - details below.

Dylan William interview here.

Klassen et al. (2011) critical review of much of the CTE research.

Professor Nazım Çoğaltay kindly responded to my question about his view of Hattie's use of his research,
"The interpretation is completely subjective and contrary to the spirit of meta-analysis studies. However, my study was done in 2017. So it is a quantitative combination of the studies done until 2017. Eels' work is from 2011. Of course, there will be an intersection set in the works we combine. However, each researcher will discuss his own style in his research while doing meta-analysis. Therefore, interpretation of the correlation value was not correct without understanding how I did it."
Prof Çoğaltay made it clear he did not want his comments to be used polemically.

I responded that I have put together a summary of a large number of peer reviews, that form a strong critique of Hattie's methods, but not polemical. 

Given the dominance of Hattie's views in Education Policy, and his claim that CTE is the #1 influence in education, more than 3 times more effective than family life, socioeconomic status & student motivation, I believe a strong critique is necessary and warranted.

Summary of the Peer Review

Hoogsteen (2020 & 2020b) argues that the correlational nature of studies means we can not ascribe improved achievement to CTE, since it is equally likely that causation could be the reverse or bi-directional, i.e., Achievement increases CTE. He also casts doubt on the validity of the different CTE questionnaires. He concludes,
"...although results of meta-analysis have led collective efficacy to be named the top influence on student achievement, this review has shown that meta-analyses may not be the most reliable format to, especially as it relates to CTE, make this claim. In contrast to the comments by Donohoo (2018) that policy makers, system, and school leaders should aim to develop collective efficacy throughout reforms, maybe in light of everything presented in this review, school and system leaders should instead take heed to the words of Wiliam (2016) and not put a heavy emphasis on meta-analysis to identify priorities for the development of teachers. In doing so they risk spending a great deal of time helping teachers improve aspects of their practice that do not benefit students." (p. 583-584)
Wiliam also confirms this problem with correlational studies and attributing causation to CTE and suggests the relationship is reverse, i.e., Improved Achievement leads to CTE.

Klassen et al. (2011) conclude,
"This review investigates the state of teacher self- and collective efficacy research conducted from 1998 to 2009... Continuing problem areas were a lack of attention to the sources of teacher efficacy, continued measurement and conceptual problems, a lack of evidence for the links between teacher efficacy and student outcomes, and uncertain relevance of teacher efficacy research to educational practice..." (p. 21)
Zhao (2019) concludes,
"Measurement issues and congruence with established theory are a severe problem affecting collective efficacy research (Klassen et al., 2011). The illegitimacy of the teacher collective efficacy measurements cast a shadow on their results and might lead to inaccurate conclusions of the research." (p. 80)
Eells herself, confirms the problems of correlational studies,
"Correlational research can only go so far... correlation is not sufficient to determine causation... but this study is limited because it cannot address causation." (p. 122)
Is CTE More than 3- Times more Powerful than Socioeconomic Status?

Despite the caution of the many peer reviews, Hattie & Donohoo (2018b) continue to make significant causation claims using this 1-Eells PhD study. 
"collective teacher efficacy is greater than three times more powerful and predictive of student achievement than socioeconomic status. It is more than double the effect of prior achievement and more than triple the effect of home environment and parental involvement. It is also greater than three times more predictive of student achievement than student motivation and concentration, persistence, and engagement."
The Eells PhD

Eells reports an average correlation r = 0.617 between Collective Teacher Efficacy (CTE) and Student Achievement. (p. 111)

Hattie then converts this to an effect size of d = 1.57. But there are major problems with this conversion, and reputable evidence organisations do not accept correlation studies as good evidence (e.g., EEF & WWC) - See Correlation for details.

Bergeron & Rivard (2017) in their critique of Hattie, published an example of the danger of converting a correlation to an effect size:

The beneficial effects of ice-cream on intelligence - a delicious correlation (r = 0.7): 

If we convert r to an effect size = 1.96! 

Much larger than for Hattie’s Collective Teacher Efficacy!


Many peer reviews, Lipsey et al. (1993, 2012), Bakker et al. (2019) the problem of converting correlation into an effect size, e.g., Kraft (2018),
"Effect sizes from studies based on correlations or conditional associations do not represent credible causal estimates."
What is Collective Teacher Efficacy?

Eells definition (p. 8),
"Collective Teacher Efficacy is an emergent group level property referring to the perceptions of teachers in a school that the faculty as a whole will have a positive effect on the students (Goddard, Hoy, & Woolfolk Hoy, 2000)."
How was CTE measured?

Eells reports by 2 methods both self-report questionnaires - the CE scale and the CTBS.

The CE questionnaire, from p. 155:
The CTBS questionnaire, from p. 156:

Eells explains these self-reports are (p. 75),
"...then aggregated so that the school yields one score."
Eells also reports a major problem with using 2 different measures (p. 109),
"Another important moderator was the tool used to measure collective teacher efficacy. Studies that used the short form of the CE-SCALE ... yielded higher effect sizes than studies that used the CTBS..." 
Hoogsteen (2020) also discusses this in his critical review, (note Hoogsteen uses the Correlation (r) and calls this the effect size, but it is a different to Hattie's effect size (d),
"...Studies that used the long version of the Collective Efficacy Scale had an effect size of .605, the short form produced an effect size of 0.645, those that used the Collective Efficacy belief scale had an average effect of 0.464, and studies that used a different measure had an effect size of .455..." (p. 583).
Converting these correlations (r) to effect sizes (d), which as discussed before, is in itself problematic, we get the wildly different effect sizes of: 1.52, 1.69, 1.05, 1.02

Hattie & Donohoo (2018b) cite Tschannen-Moran & Barr (2004) who also discuss the importance of the questionnaire, yet Hattie & Donohoo fail to mention this,
"A new measure of collective teacher efficacy was developed for this study because of concerns that the existing measure developed by Goddard et al. (2000) artificially drives down the collective efficacy scores of schools in more challenging environments by its explicit measurement of task difficulty." (p. 199)
Hoogsteen (2020) discusses the problem of self report,
"The one-time data collection in these studies presents the limitation of only measuring the relationship between self-efficacy and collective efficacy at one point in time and does not measure changes in either variable. Furthermore, the studies’ data were confined to self report surveys, and as Cansoy and Parlar (2017) admit, respondents’ appraisals of their collective efficacy may be highly influenced by their self-efficacy, particularly those in key managerial and leadership roles. Currently, at best, one can claim that self-efficacy and collective teacher efficacy co-vary, meaning when one is strong, so is the other, and vice-versa." (p. 579)
Zhao (2019) also discusses the problem of the different questionnaires used to measure CTE,
"...there is the measurement issue of collective efficacy. The most commonly used collective teacher efficacy measures are variations of Goddard et al.’s 21-item Collective Teacher Efficacy Scale or its revised 12-item short version. Goddard’s work has laid the foundation for collective efficacy research and made an outstanding contribution to the development of this research area. Nonetheless, Klassen et al. (2011) pointed out, in the literature review of teacher self- and collective efficacy research from 1998 to 2009, that some of the content of Goddard’s measures 'displays a lack of congruence with theory’' (p. 35). Several items were orientated toward external determinants, and others focused on teachers’ current abilities rather than on the 'more theoretically congruent forward-looking capabilities' (p. 35).

The scale Somech and Drach-Zahavy (2000) used in their study included items that was incongruent with efficacy theory, such as 'The teachers of this school have excellent job skills' and 'Team teachers that can perform their jobs as well as this team are rare' (p. 653), which are also present focused. However, a few measures investigated in Klassen et al.’s review were more congruent with efficacy theory. For example, several studies used Tschannen-Moran and Barr’s (2004) collective efficacy scale, a 12-item scale focusing on teachers’ collective capabilities, e.g., 'How much can teachers in your school do to produce meaningful student learning?' (p. 196), displaying a closer congruence to collective efficacy theory." (p. 79)
Hattie's Recent New Interpretation & Presentation of CTE

In his Corwin webinar (mid 2021) Hattie changed the influence "collective teacher efficacy" to "teachers working together as evaluators of their impact".

Hattie's presentation does not faithfully represent the original studies and their definition of CTE.

As you will see in the other analyses of Hattie's work, this type of misrepresentation dominates Hattie's presentations.

How was Student Achievement Measured?
"School achievement is typically measured with standardized tests, but studies that use other measurements were included..." (p. 76).
An example of the range of tests used, from p. 92, note the timing of CTE measure is also important, as Wiliam discusses below.

Bergeron & Rivard (2017) warn of the disparate measures of achievement that Hattie averages together,
"Hattie talks about success in learning, but within his meta-analyses, how do we measure success? An effect on grades is not the same as an effect on graduation rates. An effect on the perception of learning or on self-esteem is not necessarily linked to 'academic success' and so on. A study with a short time-span will not measure the same thing as a study spanning a year or longer. And, of course, we cannot automatically extend observations based on elementary school students to secondary school or university students."
Quality Control

Eells's flowchart from p. 79,

This flowchart is useful as it displays the protocol for meta-analyses and also illustrates the reason why many scholars (e.g., Shannahan (2017) & Wecker et al. (2017)) have asked Hattie to remove duplicate studies from his averages.

Eells also details another criterion (p. 75),
"Any studies conducted at the post-secondary level were excluded, since a college education is not compulsory, and the student population is comprised of self-selected individuals." 
Does CTE cause High Achievement, or does High Achievement cause CTE - Bi-directionality?

Many of the studies Hattie cites also raise this issue. However, Hattie never discusses this. Although, Hoogsteen (2020) reports, 
"...Donohoo (2018), notes that caution should be taken regarding the directionality of included variables because causal direction was determined in advance. This is the case in many studies, and in many instances conclusions are drawn that can be misleading. For example, both Bandura (1993) and Goddard (2001) investigated and confirmed the link between collective teacher efficacy and student achievement, however, in both cases achievement was measured prior to CTE." (p. 576)
Also, Hoogsteen (2020) & (2020b), & Wiliam (2019b) mount strong arguments for bi-directionality or the opposite effect.
"It's not teacher collective efficacy that causes high achievement, it's high achievement that causes teacher collective efficacy." (Wiliam (2019b))
Dylan Wiliam interview here-

So the timing of the CTE questionnaire, before or after the school rating on achievement, is significant. Yet, none of the studies control for this variable.

I've just witnessed Tom Brady and his team win the Super Bowl in 2021. If we could have measured their Collective Efficacy before the Win as compared to After the Win, and also measured the team that Lost, I wonder what that would have told us?

The Bi-directionality of CTE is also a key component of the Hoogsteen (2020) critical review,
"The concept of bi-directionality as it relates to collective teacher efficacy is illustrated in Tschannen-Moran & Barr (2004) when they state that there is a reciprocal relationship between collective teacher efficacy and student achievement. The relationship is such that the school environment can affect teachers’ belief in their collective efficacy to improve student achievement and increased student achievement can increase teachers’ sense of collective efficacy (p. 196). Eells (2011) proffers a similar explanation when she notes that CTE influences cultural norms when belief leads to action, and cultural norms influence CTE when action transforms belief (p. 65). In fact, inquiry into the directionality of collective efficacy can be found in oft-studied areas linked to CTE, student achievement, teacher outcomes, leadership, and professional learning communities. It should be noted, definitive evidence regarding directionality has not been found." (p. 576)

Hoogsteen (2020) concludes, 

"...conclusive evidence has not been found regarding the directionality of collective efficacy, no matter the area of study relating to CTE. Differing results and/or inconsistencies because of study methodology and timing of data collection have occurred. Even so, what has often happened is that researchers have made strong conclusions that are only partially supported because of the correlational nature of the studies , and the inconsistencies have led them to theorize and infer about reasons for their results so that they align with social cognitive theory. With that said, school and district leaders as well as policy makers should take a cautious approach when seeking to develop collective teacher efficacy as a primary strategy to increase achievement, improve outcomes such as innovation and behavior management, implement professional learning communities, and the other school-related outcomes discussed in this section. Hattie and most of the researchers he cites, use the theoretical work of Bandura (1986, 1997) who postulated four sources of efficacy-shaping information: mastery experience, vicarious experience, social persuasion, and affective state." (p. 580)
Implementation in Schools

The "cult of the guru", Eacott (2017) dominates my jurisdiction, Melbourne, Australia. Hattie is authoritative in the daily lives of over 50,000 teachers. Lesson plans, Annual reviews, PD & Interviews must all revolve and link to Hattie's book, Visible Learning - see HITs.

Our latest Ed. Dept initiative is to train all (1000+) Principal class teachers in CTE as defined by Hattie & Donohoo.

Hoogsteen (2021) provides wise advice on the implementation of CTE in Schools, 

"As they begin, leaders should survey their staff to gauge initial collective efficacy beliefs. Using an instrument such as those found in Goddard et al. (2000) or Tschannen-Moran and Barr (2004) would be ideal. Measuring initial levels of enabling conditions being applied using the survey developed in Donohoo et al. (2020), known as the enabling conditions for collective efficacy scale, would also be useful. School leaders should also decide which outcome they want to focus on, teacher behavior outcomes or student achievement, or both. Achievement data is usually readily available so collection of such should be relatively easy. On the other hand, teacher behavior data would be considerably more difficult to gather. A self-report survey instrument could work, but also some sort of observational data would likely be needed. Even so, tracking changes over time in teacher behavior would be challenging. Collecting the same data again at pre-determined intervals over time would help determine if efforts to improve collective efficacy resulted in changes in CTE and if such changes influenced teacher outcomes and/or achievement.

Although action research of this nature would be time consuming and out of the area of expertise for many school leaders, it is probably the best way to know if collective efficacy can be manipulated, and if such manipulation results in significant changes. Studies of this nature are required to move the knowledge base related to collective efficacy forward, and if studies such as those suggested are conducted the results may then be useful and applicable to school leaders. If a case study or action research approach to answering the questions raised by this review turn out to be too difficult to conduct in practice, it may be time to cease viewing collective efficacy as a shaper of the normative environment of a school (Goddard et al., 2000) and a characteristic to build. Instead, school leaders and researchers may need to view CTE as an indicator of the health of the organization and result of having implemented and performed effective school practices." (p. 83)

However, Hoogsteen (2020) also warned,
"As Loughland and Ryan (2020) note, with this declaration comes an uncritical adoption of the collective efficacy bandwagon when effect sizes are used in support of professional learning consultancies that are motivated by profit." (p. 575)
Professor John O'Neill (2012) also wrote a timely warning, about Hattie's influence on Education policy and his financial interest in the solutions proposed:
"The discourse seeks to portray the public sector as ‘ineffective, unresponsive, sloppy, risk-averse and innovation-resistant’ yet at the same time it promotes celebration of public sector 'heroes' of reform and new kinds of public sector 'excellence'. Relatedly, Mintrom (2000) has written persuasively in the American context, of the way in which ‘policy entrepreneurs’ position themselves politically to champion, shape and benefit from school reform discourses." (p. 2)
Also, McKnight & Whitburn (2018) continue the debate on conflicts of interest,
"The Visible Learning cult is not about teachers and students, but the Visible Learning brand. It is not dialogic, it brooks no argument and is sedimented and corralled by its trademarks and proprietary symbols. As we comply, we wonder if we should; assent and unease intertwined are the defining reactions to neoliberalism’s imperatives. Educators need to be alert to affect here, and to what it may mean... 
Teacher professionalism requires 'working with colleagues in collaborative cultures of help and support as a way of using shared expertise to solve the on-going problems of professional practice, rather than engaging in joint work as a motivational device to implement the external mandates of others' (Hargreaves & Goodson, 1996, p. 20). Visible Learning demands the latter, and not coincidentally, also requires the consumption of the artefacts, including the professional development sessions, the books, the websites, and the videos. These almanacs, with their tips for teachers, offer quick fix solutions without addressing the crises of education in an increasingly complex world (Slee, 2011). In the case of the Visible Learning brand, certain teachers become licenced as fans, for example in Hattie’s collected case studies of impact (2016).  
Hattie has effected the shift from intrinsic accountability to extrinsic accountability, negating teachers’ awareness that “professional knowledge is provisional, unfinalizable, culturally produced and historically situated” (Locke, 2015, p. 82). It cannot meaningfully be reduced to a list of strategies." (pp. 12 ff)

The Quality of the Studies

Professor Ewald Terhart (2011):

"It is striking that Hattie does not supply the reader with exact information on the issue of the quality standards he uses when he has to decide whether a certain research study meta-analysis is integrated into his meta-meta-analysis or not. Usually, the authors of meta-analyses devote much energy and effort to discussing this problem because the value or persuasiveness of the results obtained are dependent on the strictness of the eligibility criteria" (p. 429).
Many scholars, e.g. Prof John O' Neill (2012a), have been asking for a long time now, that Hattie remove studies that used University students. 

Schulmeister & Loviscach (2014) Errors in John Hattie’s “Visible Learning”.
"Many of the meta-analyses used by Hattie are dubious in terms of methodology."
Nielsen & Klitmøller (2017) also question Hattie's "quality",
"Hattie is not directly concerned about the quality of his own investigation. 
In some selected contexts nevertheless, Hattie does throw out studies based on quality, but this neither consistent nor systematic" (p. 10 translated from Danish).
Nielsen & Klitmøller's criticism is based on Hattie sometimes using the following protocols to justify exclusion of meta-analyses,
"mainly based on doctoral dissertations, ..., with mostly attitudinal outcomes, many were based on adult samples ... and some of the sample sizes were tiny" (VL, p. 196).
Prof Georg Lind (2013) confirms this and also uses more examples from VL, pp. 196 ff. Where, he accused Hattie of disregarding studies that do not suit him, e.g. kinesthetic learning.

Hattie has just ignored these problems.

The Story, The Story...?

Eells theoretical construct of CTE contradicts Hattie's claim that educational influences can be isolated into separate influences. She quotes Chubb (1988) who called for a school level focus, (p. 53),
"School performance is unlikely to be significantly improved by any measure or set of measures that fails to recognize that school are institutions- complex organizations composed of independent parts, governed by well established rules and norms of behavior, and adapted for stability. Their influence on learning does not depend on any particular educational practice, on how they test or assign homework or evaluate teaching, but rather on their organization as a whole, on their goals, leadership, followership, and climate..."
Eells goes further (p. 54),
"His research found that high and low performance schools differed very little in classroom practices, and formal structures were very similar. Distinctions were evident in student bodies and informal organization: when leadership is strong, expectations are high, and authority is delegated to the classroom, students perform at higher levels. In 1989, Newmann, Rutter, and Smith found that teacher efficacy, aggregated to the collective level, is influenced by school organizational features, such as responsiveness of administrators, teacher collaboration, encouragement of innovation, and orderly student behavior."
In addition, she quotes Bandura (p. 55),
"Bandura (1993, 1997), recognizing that academic progress in a school is not only a reflection of the sum of individual contributions but also comes from the ways in which the teachers work together, measured faculty belief in the ability of the school to achieve success. This “intermediate level of independence” (Bandura, 1997, p. 248) comes from the collective responsibility for education and hierarchical building of new learning on that from previous grades. Bandura conducted a path analysis and found that a collective sense of efficacy among a faculty contributes significantly to academic achievement (Figure 6). In fact, it was a more powerful predictor than socioeconomic status, and as powerful as prior academic achievement."
Limitations of this research

Eells details a number of limitations of her synthesis (p. 122) which are echoed regularly by the peer reviews of Hattie's work,
"A related concern is the apples and oranges issue: the mix of studies synthesized may be too dissimilar to yield meaningful overall results (Lipsey and Wilson, 2001). Within this meta-analysis, this problem became apparent when examining how the variables were defined. Student achievement was operationalized in numerous ways, and CTE, in a few ways. Additionally, there was some discrepancy concerning the timing of the measures. As previously stated, although the component studies addressed the correlation between CTE and achievement, some were clearer when distinguishing prior achievement from subsequent achievement. This posed a problem because that inconsistency reduces the certainty of any conclusions drawn about how CTE can predict achievement. 
Another limitation for this meta-analysis was the small number of studies sampled. This line of research is less than 20 years old, and relies on a handful of studies to demonstrate the relationship in question... 
Correlational research can only go so far. The effect size for this meta-analysis came from Pearson product-moment correlations, which quantify the strength of a relationship. While there will always be a correlation between cause and effect, correlation is not sufficient to determine causation. This research makes it clear that CTE and school achievement vary together: They are strongly and positively correlated, but this study is limited because it cannot address causation."
Then finally on p. 124,
"The confounding of moderator variables was a particular limitation for this study."
Hattie & Donohoo continue to make Questionable Claims!

Given the limitations of this study and the many other issues raised, Hattie & Donohoo (2018b), continue with troubling claims,

"collective teacher efficacy is greater than three times more powerful and predictive of student achievement than socioeconomic status. It is more than double the effect of prior achievement and more than triple the effect of home environment and parental involvement. It is also greater than three times more predictive of student achievement than student motivation and concentration, persistence, and engagement."
Donohoo's presentation with Corwin - click here.

Donhoo, in the above presentation, cites heavily from Sandoval et al. (2011). A closer look at this study is revealing.

Firstly, Schools were rated as Exemplary or Academically Unacceptable. (p. 12)

Only 6 out of 10 schools accepted to take part in the study, but we are not told how many of those schools were rated Exemplary or Academically Unacceptable. (p. 12)

A total of only 113 teachers volunteered to complete the Collective Teacher Efficacy (Short Form) which gave the researcher 53.1% participation rate for all campuses. (p. 12). So a large element of bias is introduced into the study.

The teachers were then rated as coming from an Exemplary (46) or Academically Unacceptable (67) school. So the teachers new their rating BEFORE CTE questionnaire.

The voluntary & low participation rate and the use of the short form questionnaire of this study, raises major doubt on the extent to which Donhoo extrapolates these results to the whole School system.

Doubt on the comparison with Socio-economic status

Tschannen-Moran & Barr (2004) also cast doubt on Hattie's & Donhoo's comparison & ranking,
"Socioeconomic status did play a role as expected in explaining student achievement in the schools in this study. A significant negative relationship was evident between student achievement and the schools' percentage of students receiving free and reduced-price lunch in a school in math, writing, and English. The higher the proportion of students receiving subsidized meals, the lower the student achievement on the SOL tests." (p. 202)
"When controlling for socioeconomic status, collective teacher efficacy made a significant indepen­dent contribution to the grade 8 writing SOL; however, it did not make a significant contribution in explaining math and English achievement." (p. 202-203)
Hattie Continues to Rank

Hattie in his collaboration with Corwin promotes efficacy as the new Number 1. Yet, a couple months previously, in an interview with Lovell (2018), he said his rankings were misleading and he does not rank anymore - click here for interview.

Another Hattie presentation where he says Collective efficacy effect size of 1.57 is the highest we know.

Examples of Different Definitions of Collective Teacher Efficacy

Carmel Patterson (2021),
"Teacher collective efficacy (CE) is often misrepresented as collaborative task generation or cooperative marking or reviewing of student work. CE entails the work of teaching teams to garner collective contributions for:

-developing understanding of observations on learning,
-critiquing task requirements,
-assessing student work samples,
-creating reasoned strategies to implement and evaluation in context,
-expanding and clarifying individual teacher thinking with colleagues, and
-collectively developing practice in context."

No comments:

Post a Comment