Economists do a pretty bad job, in general, of being careful when making what the literature calls multiple comparisons. That term encompasses situations where, for example, the researcher has quite a lot of dependent variables and is considering impact estimates for all of them. Of course, if there are 100 independent outcomes, then we would expect statistically significant impacts on five (or ten depending on your alpha) of them even in the world of the null where the treatment has no effect on any of them.
The social science and statistics literatures contain a number of ways to address this issue somewhat formally, including various procedures for adjusting significance levels, such as the Bonferroni adjustment, and dimension reduction techniques applied to the outcome variables.
This recent report by Peter Scochet of Mathematica for the Institute for Education Sciences provides a useful introduction to the (surprisingly tendentious) literature.
Who was my favorite student this term?
7 years ago
