The paper was linked to by both Greg Mankiw and Marginal Revolution, in both cases without comment, which links presumably led to all the attention.
For example, Rush Limbaugh hyped the paper as confirming his own reasoning based on what one might charitably call lay theory. Limbaugh has some trouble sorting out where Conley and Dupor actually work though. Perhaps his assistant was having an off day.
MR also linked to this critique by Michigan gradual student Noah Smith. Noah's critique has two main bits. First, he emphasizes the lack of a statistical difference between the point estimates in the paper and zero. True enough, but he edges a bit too close for my taste to equating "not statistically different from zero" with "equals zero". The point estimate is still the best estimate in the sense that it is the solution to the optimization problem embodied by the estimator. Yes, it is imprecise, and that is important when thinking about how to update one's beliefs about the effects of the ARRA, and yes, it is not statistically different from zero. At the same time, it is very different, and perhaps even statistically different, from various positive estimates of ARRA employment impacts offered up by, for example, the administration. In my view, as a casual Bayesian, the effect of the Conley and Dupor should be to add additional uncertainty to claims of large positive impacts made by others.
Noah's other critique is aimed at Greg Mankiw, for linking to the paper without comment or critique and for not linking to another, related paper. I think Greg would be liable to valid criticism if he had hyped the paper, but to me just linking to it means "hey, this paper by two reasonable economists looks interesting but I've been too busy to really dig into it yet". I do not, in general, link to papers I have not closely read on my blog, but it does not seem to me unreasonable to provide a link-without-comment with the intention of starting a discussion, just as Tyler Cowen did at MR.
Paul Krugman (surprise!) does not like the paper.
Early on, he has this to say:
Remember, the stimulus was not big compared with the economic downturn. The original Romer-Bernstein estimate was that it would, at peak, reduce unemployment by about 2 percentage points relative to what it would otherwise have been. And most of that effect was supposed to come through measures that would have been common to all states: tax cuts, transfer payments, etc.. At most, differences between predicted effects among states should have come to no more than a fraction of a percentage point off the unemployment rate.
I am not quite sure what Krugman has in mind with this paragraph. I think it means that he does not fully understand how instrumental variables work their magic - not surprising perhaps given his background as a trade theorist. The point is to find a variable, the instrument, that isolates a bit of exogenous variation in the independent variable of interest, in this case stimulus spending. The instrumental variables procedure isolates this exogenous variation and determines its effects. In a common effect world, wherein every dollar of stimulus spending has the same impact on employment, and it is that world in which this literature and this paper operate, all you need is to then appropriately scale the instrumental variables estimate to get the full impact of the stimulus spending. For consistency of the estimates, it does not matter that the fraction of the variance in spending pinned down by the instrument is small, as long as the instrument clearly predicts stimulus spending. Where the fraction of the variance explained by a valid instrument shows up is in the standard errors and they are large here, as one would expect.
Krugman next presents a bar graph showing before-after changes in state unemployment rates in a bar graph and then adds:
To tease any effect of the stimulus out of these interstate differences, if it’s possible at all, would require very careful and scrupulous statistical work — and we’d like to see some elaborate robustness checks before buying into any results thereby found.The latest anti-stimulus paper shows no sign of that kind of care. It makes no effort to control for the differential effects of bubble and bust. It uses odd variables on both the left and the right side of its equations. The instruments — variables used to correct for possible two-way causation — are weak and dubious. Dean Baker suspects data-mining, with reason; the best interpretation is that the authors tried something that happened to give the results they wanted, then stopped looking.Really, this isn’t the sort of thing worth wasting time over.
Unfortunately for the reader trying to engage with the Conley and Dupor paper, Krugman does not say what variables he thinks are odd. Is employment odd? Looking at employment rather than unemployment - which may or may not be the alternative Krugman has in mind - seems reasonable enough as impacts on employment capture effects on the number of discouraged workers, while impacts on unemployment rates do not. Krugman similarly does not bother to explain why the instruments are dubious, he just asserts it. Conley and Dupor make a positive case - see Section 3.1 - for their instruments in their paper; surely Krugman can be expected to make a negative one in his response. Certainly the instruments can be questioned; really compelling instruments are essentially non-existent in macro. Why not make the case? Worst of all is Krugman's claim that the instruments are weak, which is technical shorthand for saying they do not have a strong (enough) relationship with stimulus spending. This claim is simply wrong, as shown in Table 3 in the paper.
In short, Krugman's response disappoints the serious reader. Of course, the fact that Krugman's response is weak does not mean that the Conley and Dupor paper is worth paying attention to; it just means that one must look elsewhere for serious discussion.
I happened to be at Western Ontario on Wednesday for a conference and had a chance to talk to Tim about all the craziness surrounding the paper. He said that he had, as of that time, gotten about 60 pieces of hate email, as well as lots of media inquiries, almost all of which he had declined. I should note, too, that it was Tim who emphasized to me the importance of comparing the estimates to values other than zero and who pointed out that employment is very much not an odd dependent variable.
Full disclosure: I overlapped with both Tim and Bill in gradual school at Chicago, though we were not close friends. I skimmed the paper when writing this post but have not read it closely due to having spend the whole week (other than Monday) at conferences.
'the best interpretation is that the authors tried something that happened to give the results they wanted, then stopped looking.'
ReplyDeleteI've never heard an economist say anything like this before, it seems way beyond the pale. I don't think Krugman should be referred to as an economist any more.