Science (the thing) and Science (the journal)

David McKenzie on a new study from some non-economists in Science that purports to show that teachers do not matter and methods do.

I am stunned that this got published in Science. The fact that it did has led to a negative revision in my beliefs about the quality of the research it lets through its editorial process. Though even before this, I would likely have applied a somewhat weakened version of this theory of social science publications in medical journals to Science.

The key here is that the dispersion of knowledge regarding study design (or identification, if you prefer) among practicing researchers, and thus among editors and referees, is, on average, much weaker outside of economics. Of course, many disciplines have some researchers who are very strong in these areas, and, of course, other disciplines that are relatively weak on study design often know lots of useful things that most economists do not (item response theory, anyone?). The point is that disciplinary differences in the relative dispersion of knowledge about study design provide an incentive for weakly designed studies to find their published homes outside of economics. It would be a wonderful world indeed if journalists would adjust their reports for this important selection problem.

At the same time, I have some sympathy for the point of the paper. For many, but not all, subjects, lectures should play only a modest role in the teaching process. I think universities produce too many lectures and too few hands-on sessions in many cases. In econometrics, for example, there is simply no substitute for sitting down in front of the computer and confronting the data with the tools, and then writing up the results of the confrontation. Lectures have a role, but practice, made as realistic as possible, is a necessary complement.

Finally, McKenzie worries that economists do not have an outlet for such "exploratory" work. First, as a not irrelevant aside, Science is probably not a good outlet for exploratory work in any field. Second, I think economists do have ways to circulate or otherwise make use of exploratory work. One natural path is for a researcher to use it as a lever to get funding for confirmatory work that is better designed and has larger sample sizes. Another approach available to those with access to the standard working paper series such as the NBER and the IZA, is to circulate exploratory work in working paper form with no intention of eventual publication. There are also blogs, many of which function as a platform for the discussion of research at the working paper stage. Finally, one might have thought that a journal such as Economics Letters, or perhaps unrefereed conference volumes, would play this role.