Causal follies: The Atlantic on Citizen Schools

The Atlantic writes about a very PR-friendly program in which corporate employees teach after regular school hours in middle schools serving disadvantaged students.

Sounds great. Surely someone has evaluated it in some serious way, as the article indicates that it is expanding around the country. What does the Atlantic writer offer up on this score?

First, we learn that students in the program are self-selected:

Citizen Schools now serves about 5,000 middle school students each year across eight participating states, and the program documents lasting effects for its participants. Kids who have gone through Citizen Schools in middle school are less likely to be absent from high school, and graduate from high school at a rate 20 percent higher than their peers.

No causality there. Is there anything else? Well, there is this bit:

Michael Andrew says he knows participating in the program had a positive effect on him. When he was a fourth- and fifth-grader in Boston, he was one of the earliest enrollees. Now, he’s a 24-year-old graduate of Syracuse University who works at AllianceBernstein in information technology. And he’s back with Citizen Schools, this time as a volunteer.

That would be an anecdote, which is to say a participant evaluation with n = 1. The literature provides no real support for such participant evaluations.

Now, the sad fact of the matter, is that there is no way for the reader to be sure that there is not a serious evaluation out there that the Atlantic reporter missed (or did not know to look for) or whether this is really all the literature offers in the way of support for the program. Either way, a disappointment.

Sigh.