Mentoring

Effect Size d = 0.15  (Hattie's Rank = 120)

Hattie averaged 2 meta-analyses - 0.13 & 0.16 to get 0.15

In previous lectures, Hattie has called Mentoring a disaster! 2008 Nuthall lecture: 

Details of the 2 Meta-Analyses Hattie Used

1. Eby et al., Does Mentoring Matter (2007).

Eby lists over 60 different correlations. Hattie does not give details of how he got his effect size of d = 0.16. It seems that he has averaged the correlations of ALL mentoring outcomes, which includes NON-performance outcomes (substance use, stress, motivation, interpersonal relations, etc).

Also the "withdrawal" & "psychological stress" outcomes are framed in a negative manner, so the negative correlations means withdrawal behaviour decreases with increased mentoring. So if Hattie has averaged all of these outcomes, the withdrawal correlations need to be changed to +. Note this has been a problem with Hattie's interpretations in other categories, e.g class size and feedback.

The ONE correlation, with regard to ACADEMIC ACHIEVEMENT, is r = 0.19 which can be converted into an effect size d = 0.39 (see Correlation) which is more than TWICE the effect size that he reports! See table below from p. 262.

Professor Eby concludes: 

"academic mentoring has stronger associations with outcomes than does youth mentoring" (p. 263).

Also, Eby reports several limitations: given the correlation nature of many of the studies, our findings do not provide unambiguous evidence that mentoring causes outcomes. Rather, this leads to more controlled designs: control groups, random allocation, and controlling for TIME! (p. 265)

I contacted Professor Ebby, and she explained another confounding variable - the time students are mentored. In this study, she just considered whether a student was mentored or NOT. In later studies, (see below) she found 

"stronger effects when examining the amount of mentoring support provided."

2. Dubois et al., (2002). Effectiveness of Mentoring Programs for Youth.

Similar to Eby et al., Dubois et al., detail a number of effect sizes. 

However, this time it appears Hattie does choose the academic outcome - see table below (from Dubois, p. 176). Hattie reports an effect size d=0.13.

DuBois et al conclude: Inferences regarding the influence of different variables are tentative because of the correlational nature of the studies. Better controlled studies reveal higher effect sizes. (p. 191)

Eby's Later More Detailed Study

Eby et al., (2013) An Interdisciplinary Meta-Analysis of Potential Antecedents, Correlates, and Consequences of Protege Perceptions of Mentoring.

Eby et al., focus on the amount of time for mentoring and get an average correlation (for learning) of r = 0.26 when converted gives d = 0.54 (p. 458). Hattie does not cite this study (@August 2022).

Eby et al., state 

"there is substantial variability in the nature and quality of interactions among mentors..." (p. 444).

The following diagram shows the complexity and variability involved (p. 459):

Once again a single effect size number misrepresents the complexity of the influence. Eby et al., conclude:

"The results of this research must be viewed within the context of the limitations associated with the literature that comprised this review. The vast majority of effects reported were based on cross-sectional data (93%) [i.e., correlational studies] and none were based on data from experimental designs, precluding our ability to make causal inferences" (p. 463).

Hattie's Analysis (Visible Learning, p. 187-188)

'Mentoring had a close to zero effect on performance outcomes (d = 0.08), although there were higher effects on attitudes (satisfaction d = 0.6, school attitudes d = 0.19), and on motivation and involvement (d = 0.11) (Eby, Allen, Evans, Ng, & DuBois, 2008). That is, there is more change on attitudes than achievements, probably because “attitudes are more amenable to change than are outcomes that are more contextually-dependent” (p. 16).

It was the case that effects were higher for academic mentoring than for youth (at risk, family-related mentoring) and workplace mentoring.

DuBois, Holloway, Valentine, and Cooper (2002) investigated many outcomes from mentoring. Across their 575 effect sizes, the average was d = 0.18 on achievement, and these low effects occurred when the program was one-on-one or in groups; the effects were lower in schools than in workplaces and higher for trained compared with nontrained mentors, but there was no relation with the frequency of contact nor the length of relationship between mentors and youth. The effects were similarly low for emotional or psychological outcomes (d = 0.20), problem and high risk behaviors (d = 0.19), social competence (d = 0.16), and career and employment outcomes (d = 0.19).'

NOTE- It is difficult to find how Hattie got some of these effect sizes, e.g he quotes Eby et al., ES=0.16, but in this quote ES=0.08. But, if you average the 3 performance outcomes you get an average r= 0.10 which when converted gives ES=0.20.

Other Organisations on Mentoring

There are organisations who show a lot of evidence for the impact of mentoring e.g., http://www.mentoring.org