Multiple comparisons

The multiple comparisons problem arises when doing large numbers of statistical tests. For example, one might estimate the impact of some treatment on 100 different outcomes. Ignoring correlations among the outcomes, in a world where the population treatment effect equals zero for all 100 outcomes, one would still expect five estimates to be statistically significant at the five percent level.

xkcd provides a humorous illustration of the problem.

My friend (and fellow Heckman student) Peter Schochet of Mathematica prepared a very nice (and very accessible) survey of the related literature for the Institute of Education Sciences.