Monte Carlo analyses of matching and weighting estimators

I happened to read this paper earlier this week:

How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score by Martin Huber, Michael Lechner, Conny Wunsch (October 2010)

Abstract:
We investigate the finite sample properties of a large number of estimators for the average treatment effect on the treated that are suitable when adjustment for observable covariates is required, like inverse probability weighting, kernel and other variants of matching, as well as different parametric models. The simulation design used is based on real data usually employed for the evaluation of labour market programmes in Germany. We vary several dimensions of the design that are of practical importance, like sample size, the type of the outcome variable, and aspects of the selection process. We find that trimming individual observations with too much weight as well as the choice of tuning parameters is important for all estimators. The key conclusion from our simulations is that a particular radius matching estimator combined with regression performs best overall, in particular when robustness to misspecifications of the propensity score is considered an important property.

The paper presents a thoughtful Monte Carlo analysis of various matching and weighting estimators for treatment effects. Unlike most Monte Carlo analyses, which specify relatively simple DGPs, this Monte Carlo is based on real data: in this case, the sort of German administrative data used to evaluate active labor market programs. This feature makes it a useful complement to the two Monte Carlo studies by Busso, DiNardo and McCrary (2009a,b).

Two finding stood out to me. First, like the other studies in the literature, including the important paper by Froelich (2004) Review of Economics and Statistics, Huber et al. find that single nearest neighbor matching (also called pair matching) performs very poorly relative to other estimators in mean squared error terms. It really is time to put this estimator (which is one of the, if not the, most widely used estimator in practice) away in the curio cabinet of obsolete statistical procedures. Second, I liked the authors' suggestion of a new trimming method to deal with the instability that sometimes results when matching or weighting on the propensity score (the conditional probability of treatment) in cases where some scores are very close to zero. Their scheme awaits a thorough theoretical econometric treatment, but it has nice features and works well in their Monte Carlo.

This paper, and the Busso et al. papers, are all well worth reading for those interested in the estimation of causal effects using identification strategies based on "selection on observed variables" or unconfoundedness.