Replication in economics

Greg Mankiw linked to this horrifying tale of replication in one of the hard sciences.

Do economists do better?

Well, I think editors in economics are not as hopelessly incompetent as the editors described in this saga. They might be slow - certainly I am too slow - but they tend to worry about substance rather than form. In most cases, they are not this high-handed either. I have certainly never experienced anything like this and do not recall stories of editor behavior like this.

On the other hand, I know of several recent instances where junior people have replicated papers by famous people and found some problem with what was done. Some of these cases arise from a class taught by David Card at Berkeley wherein the students do a replication as their class paper.

I am going to leave names out of this on the misbehavior side, but in a couple of relatively high profile cases involving top journals, the senior people did not really distinguish themselves. In a way this is odd. Someone with tenure at a top department does not really have much to lose by being helpful and honest. We all know that empirical papers in economics are complicated things, carried out over long periods of time, sometimes in temporally separated frantic bursts of activity to meet deadlines, and often with assistance from multiple graduate student research assistants who are, rather by definition, still learning how to do research. So occasional errors should not come as much of a surprise. Moreover, for senior people, tenure is not at risk. Even reputation is not really at risk. And yet, there is sometimes trouble.

I do not know all the details of these cases, of course, and usually have more information about one side than the other, which is why I am not mentioning names, but we can, I think, do better as a discipline. Sometimes doing better just means chilling out a bit.

Let me toss out some praise, too. Rajeev Dehejia was helpful throughout the process in which Petra Todd and I replicated his papers with Sadek Wahba. At the end of the day, we still disagree in regard to the meaning of the results that emerge from the Supported Work data, but at least we agree on the results! The process does not have to be unpleasant or worse.