Genmatch

Genetic Matching for Estimating Causal Effects:

A General Multivariate Matching Method for Achieving Balance in

Observational Studies

Alexis Diamond

ŽJasjeet S. Sekhon



Abstract

This paper presents Genetic Matching, a method of multivariate matching, that uses an evolutionary search algorithm to determine the weight each covariate is given. Both propensity score matching and matching based on Mahalanobis distance are limiting cases of this method. The algorithm makes transparent certain issues that all matching methods must confront. We present simulation studies that show that the algorithm improves covariate balance, and that it may reduce conditional bias if the selection on observables assumption holds. We then present a reanalysis of a number of datasets in the LaLonde (1986) controversy.

JEL classification: C13, C14, H31

Keywords: Matching, Propensity Score, SeKeywords: Matching, Propensity Score, Selection on Observables, Genetic Optimization, Causal Inference

I happened to read this paper (for the second time) a couple of days ago. It introduces for an economist audience (the authors are political scientists) a new algorithm for the construction of estimates of causal effects based on an assumption of "selection on observed variables". Other disciplines sometimes call this assumption unconfoundedness or ignorability (and economists sometimes refer to the "conditional independence assumption"). What all this jargon means is that the researcher thinks that conditional on covariates available in the data, individuals are assigned to treatment, whether by nature, by institutions, or by their own choices, or some combination of these, in a way that is unrelated to their untreated outcomes. Essentially, one makes the case for an assumption random assignment conditional on observed characteristics, where in good papers that case consists of more than "this is all I could do" or "look at how many different variables I matched on mom!".

To see what makes the method outlined in this paper (and in more technical detail in other papers available on Jas Sekhon's webpage, including some co-authored with my UM political science colleague Walter Mebane) it helps to think about what a randomized experiment does. Statistically, random assignment of individuals into treatment and control groups balances the distributions of both observed and unobserved covariates between the treated and control units. When samples are small, this balance may be imperfect in particular realizations, but as the sample gets larger, the balance, statistically speaking, becomes better and better.

What Genmatch does is to choose untreated units to match to the treated units in an observational study based solely on a criterion of post-match balance. This contrasts with the usual approach in economics of using something like nearest neighbor matching or kernel matching on estimated propensity scores (probabilities of treatment) in an iterative process in which balance is tested at each iteration and the propensity score model is made more flexible by adding additional terms until some desired level of balance is achieved.

The paper includes two Monte Carlo analyses as well as an application to the much-abused data from LaLonde's (1986) seminal work on the National Supported Work Demonstration. The NSW application is well done and sensibly interpreted. One of the Monte Carlo analyses, drawn from the literature on matching outside of economics, has the bizarre feature that it includes matching on instruments, that is, on variables that affect participation in treatment but do not otherwise affect outcomes. As Jay Bhattacharya explains at length, you should not do this.

Readers interested in Genmatch will also likely be interested in inverse probability tilting, which has the same spirit of building balance maximization into the estimation but in the context of weighting estimators rather than matching estimators.

Atlantic Causal Inference Conference

I spent the last two days at the Atlantic Causal Inference conference which, this year, was held far away from the Atlantic at Michigan's School of Public Health.

The conference was great inter-disciplinary fun, as it included statisticians, biotstatisticians, epidemiologists and even a few economists. I got to meet some famous people from other fields and also learned a lot (though there were a couple of presentations where simultaneous inter-disciplinary translation would have helped).

Two quotes that I liked well enough to write down:

"My colleagues, they study artificial intelligence; me, I study natural stupidity" - Amos Tversky

"Matching is an attempt to approximate what reweighting is doing directly" - Justin McCrary

In addition to providing the Tversky quote, Sander Greenland's keynote speech taught me the term "nullism", which indicates excessive reverence for the null hypothesis in classical hypothesis testing. I expect that term to come in handy in future.

Causation and correlation: occupational edition

Reuters produces and the NYT publishes a report on a study of bivariate correlations between occupations and death rates. Neither the authors of the study nor the reporter seems to understand that the estimates are not causal.

This will give you the idea:

Higher than average risk of death from HIV/AIDS was found among male tailors and dressmakers and male hairdressers, while accidental poisoning by drug was high among male painters, decorators, bricklayers, plasterers and roofers.

I look forward to the pathbreaking research on the causal effect of proximity to scissors and sewing machines on HIV/AIDS that is inspired by this work.

No link to the study, no mention of whether it is published and not even any quotes from experts. This is an "F" for sure.

Hat tip: Jessica Goldberg

Stimulating counterfactuals

Greg Mankiw, with whom I usually find myself in agreement, cites the transcript of a recent news conference with President Obama and then crabs about his proposed treatment effect estimator:

Question: The American people have seen hundreds of billions of dollars spent already, and still the economy continues to free-fall. Beyond avoiding the national catastrophe that you've warned about, once all the legs of your stool are in place, how can the American people gauge whether or not your programs are working? Can they — should they be looking at the metric of the stock market, home foreclosures, unemployment? What metric should they use? When? And how will they know if it's working, or whether or not we need to go to a plan B?

Answer: I think my initial measure of success is creating or saving 4 million jobs. That's bottom line No. 1, because if people are working, then they've got enough confidence to make purchases, to make investments. Businesses start seeing that consumers are out there with a little more confidence, and they start making investments, which means they start hiring workers. So step No. 1, job creation.

The expression "create or save," which has been used regularly by the President and his economic team, is an act of political genius. You can measure how many jobs are created between two points in time. But there is no way to measure how many jobs are saved. Even if things get much, much worse, the President can say that there would have been 4 million fewer jobs without the stimulus.

So, let's suppose that Obama had just said "create 4 million jobs" without the "save" bit. Under Greg's proposed estimator we measure the number of people employed on the day of the press conference and the number employed on some future day and if the difference is more than 4 million, we say the stimulus plan worked.

I hate to be a party pooper, and maybe this sort of estimation strategy is all the rage in macro, but in addition to the obvious problem of choosing just when to measure employment in the future (i.e. picking the "after" period in the before-after estimator) is the additional problem that the assumption underlying this estimator is that without the stimulus package, the expected value of employment in the "after" period is what it was on the day of the press conference. I am certain that Greg does not really believe this identifying assumption, even though his proposed estimator relies on it. I don't believe it either.

The first lesson here is that measuring the impact of the stimulus relative to a counterfactual, which could be no stimulus or some different stimulus, is pretty hard whether the claim has to do with job creation and retention or just plain old job creation. Maybe one of those models in Greg's textbook could help out with this task?

The other lesson is that Obama, whatever his other sins, is not sinning that much worse than any other politician on this one. One thing not in short supply during the financial crisis is bold claims about difficult-to-estimate causal effects.