Another "no effect" NYC conditional cash transfer

David McKenzie summarizes the results of the Rand Corporation evaluation and justly celebrates one small victory for evidence over enthusiasm.

When random assignment yields implausible estimates

Consider the following educational treatment, as described on the Department of Education's What Works Clearinghouse:

Seventh graders were placed at random into intervention and comparison groups near the start of the school year.

Both groups were given structured writing assignments three to five times during their seventh- and eighth-grade years.

The intervention group wrote about their personal values (e.g., relationships with friends and family, religious values) and why these were important to them.

The comparison group wrote about neutral subjects, such as their daily routine, or why values they considered unimportant might be important to others.

The result? The WWC summarizes the findings:

Among African-American students, completing writing exercises about their values increased their average seventh- and eighth-grade GPA by a quarter of a letter grade (0.24 points), a change that was statistically significant. The intervention did not have a statistically significant effect on the academic outcomes of European-American students.

Among low-achieving African-American students, the effect was somewhat larger, an increase in average seventh- and eighth-grade GPA of 0.41 points. In addition, the intervention reduced the likelihood that low-achieving African-American students were assigned to a remedial program or were retained in grade.

Is this plausible? Both 0.24 and 0.41 GPA points are large impacts, particularly given that the support of the GPA distribution at many schools is largely confined to 2.0 to 4.0 or even 2.5 to 4.0.

What could go wrong? Like most educational interventions, this one is not double-blind. It may not be single-blind either, if the teachers know which students ended up in the treatment and control groups. On the other hand, given that all students wrote essays it may not have been clear to the students what the "treatment" was in some sense.

One might also worry that one of the alternative essay topics, namely "why values they considered unimportant might be important to others" is not really "neutral" and instead might have its own treatment effect. In this regard, it would be nice to have a no-essay control group along with the "other essay topic" second treatment arm. A no-essay control group has its own issues, of course, because it makes the nature of the treatment clearer to the students, but it would shed at least some light on concerns that one of the alternative essay topics may reduce GPAs rather than the other increasing them or some combination of the two.

Also, one would like a compelling theory of why such a modest treatment should have such a gigantic effect.

Interesting, and puzzling, and a good illustration of why one study is rarely definitive and, not unrelated, why good science involves both exact replications and small perturbations of provocative results.

I should have gotten one of my ECON 490 students to do their literature survey on such treatments.

Evaluating partial employment service privatization in Sweden

I get regular emails from the IFAU (a labor market policy research institute) in Uppsala, Sweden about their new working papers.

Today this one arrived:

Effects of outsourcing employment services: evidence from a randomized experiment

Helge Bennmarker, Erik Grönqvist, Björn Öckert

In many countries welfare services that traditionally have been provided by the public sector are increasingly being contracted out to private providers. But are private contractors better at providing these services? We use a randomized experiment to empirically assess the effectiveness of contacting out employment services to private placement agencies. Our results show that unemployed at private placement agencies have a much closer interaction with their placement worker than unemployed at the Public Employment Service (PES). In particular, unemployed at private agencies receive more assistance in improving their job search technology. We do not find any overall difference in the probability of employment between private placement agencies and the PES), but this hides important heterogeneities across different types of unemployed. We find evidence that private providers are better at providing em¬ployment services to immigrants, and also indications that they may be worse for adolescents. Any effects tend to fade away over time.

What makes this cool?

First, it is a social experiment conducted in a developed country other than the United States. Those are rare birds indeed.

Second, it is an evaluation focused not on the question of whether or not a given service should be offered but rather on how best to organize its delivery. In particular, the evaluation examines privatization of employment service services. The opportunities for evaluations of this sort are nearly endless, and experiments that consider how to deliver services should raise fewer political issues than experiments that deny services to a control group in order to estimate the effect of services relative to no services.

Third, the evaluators here focus not just on the final product, namely labor market outcomes, but also on the process, by reporting on impacts on things like time spent the caseworker. This helps in understanding where the impacts on labor market outcomes come from.