Meta Analysis

Great claims have been made for Meta analysis. The principle seems sound, that by combining the results of numerous smaller studies you can generate results which are both more authoritative and more general than any of the individual papers which go to make up the study. 

Professor John Hattie is a practitioner (http://www.education.auckland.ac.nz/uoa/home/about/staff/j.hattie)

I remain to be convinced by Hattie's Meta analysis. For me its a purely numeric argument. I assume that for a study to make a valid contribution to a meta analysis, the author of the Meta analysis must have read the paper. 

Hattie's meta analysis is claimed to be  a synthesis of 50,000 previous studies (This number varies in some papers on his website its as high as 180,000). If the original papers were used that would be reading 50 papers a week for about twenty years.

However Hattie does not claim to have read these studies, but to have read a much smaller number of studies summarizing previous work.  At this point one has to ask how it was possible to ensure that the literature used as input in those studies was distinct, i.e. that each of those studies did not include many of the same papers.

If you cant do this then determining just how many studies contributed to the eventual conclusion becomes problematic.

Just to make this reservation clear, I would like to state that this comment is based on over 100,000 research studies, because it is based on at least two reviews of Hattie's work, each of which were based on his work, which included the results of over 50,000....

Much of what Hattie  says is very sensible, in particular his argument that as educationalist we must move away from "it works for me"(Hattie, 1999) and take a more objective approach to educational research and model building. Statistics can be one approach to such a project. However showing that two factors are correlated is a very long way from showing that there is a causal relationship between them, and its hard to see how Hatties approach of combining Meta analysis can drill down deeply enough into the nuts and bolts of what goes on  in a classroom environment to provide us with reliable information as to what will work for us.

my 2c.

Hattie 1999 http://www.education.auckland.ac.nz/webdav/site/education/shared/hattie/docs/influences-on-student-learning.pdf