Statistics 101: a brief rant

So I spent a bunch of time over the past few days reviewing a draft final report on an evaluation that I cannot tell you about because I signed a form promising not to. Maybe I will blog about it when the results are made public and maybe not.

In any case, that is not the point. The point is that the draft had a whole bunch of instances in which an estimated coefficient not being statistically different from zero was equated with the underlying population parameter being equal to zero.

This is wrong!

Large standard errors do not mean that the population parameter equals zero, they mean that the estimate is not very precise. While that is surely disappointing, it is the very truth.

Moreover, even in cases with large standard errors, the preferred estimate - and the maximum likelihood estimate in the case of models estimated using maximum likelihood methods - is the obtained point estimate, not zero.

Classical statistics is odd in a bunch of ways (as any Bayesian will be happy to explain to you at great length) but it is what we have, more or less, so we should get it right.