The significance of significance testing

Gelman, Andrew and Hal Stern. 2006. "The Difference Between `Significant' and `Not Significant' is not Itself Statistically Significant." American Statistician  60(4): 328-331.

It is common to summarize statistical comparisons by declarations of statistical significance or insignificance. Here we discuss one problem with such declarations, namely that changes in statistical significance are often not themselves statistically significant. By this, we are not merely making the commonplace observation that any particular threshold is arbitrary—for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance. Rather, we are pointing out that even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities.

The error we describe is conceptually different from other oftcited problems—that statistical significance is not the same as practical importance, that dichotomization into significant and nonsignificant results encourages the dismissal of observed differences in favor of the usually less interesting null hypothesis of no difference, and that any particular threshold for declaring significance is arbitrary. We are troubled by all of these concerns and do not intend to minimize their importance. Rather, our goal is to bring attention to this additional error of interpretation. We illustrate with a theoretical example and two applied examples.The ubiquity of this statistical error leads us to suggest that students and practitioners be made more aware that the difference between “significant” and “not significant” is not itself statistically significant.

This article is a few years old but I just ran across it. It is a quick read, and yet one more illustration of the many conundrums that arise when one takes classical statistics too literally. I am still working on a way that I am really happy with to teach undergraduates to have a sophisticated understanding of classical significance tests.