Comparison groups for tenure letters and such

One important aspect of writing letters for scholars who have been out for a while concerns choosing a comparison group. Many tenure letter requests explicitly ask for comparisons to other scholars at the same career stage. Making these comparisons requires recalling who came out on the market when, which can be relatively easy if your department interviewed in the relevant field that year but is less easy if it did not.

Earlier this year when writing a letter for someone who received their doctorate in 2012, I was having trouble coming up with comparators, when I suddenly had an inspiration: why not compare them to the Review of Economic Studies tour class for their year? For readers less familiar with relatively obscure details of the status hierarchy in academic economics, each year this journal - the fifth of the "top five" general journals by most accounts - organizes a European tour for a set of job market stars. The 2020 tour list is here; Wisconsin made offers to two of them (and did not hire either one). The link also leads to a document with the historical lists.

As it turns out, the numbers of Google Scholar citations (as of Feburary 2020) of the seven members of the Review of Economic Studies tour for the doctoral class of 2012 (the 2011 "tour") are: 1056, 855, 678, 653, 318, 264, and 251. I was surprised by the spread in citation counts as well as the spread in publications / papers, which consulting the relevant Google Scholar pages revealed.

Embarrassingly, I opposed hiring the person at the top of the list when I was at Michigan, but I am always glad to be proven wrong when it involves a new person doing better than I expected. There are two or three other cases of similar (ex post) foolishness on my part. The fact of the matter is that a read of a job market paper and a 30 minute interview do not even come close to fully resolving the uncertainty in most cases.