Reading First

The Reading First evaluation performed by Abt and MDRC (and a cast of thousands) for the Institute of Education Sciences was finally released today. The executive summary is on the IES website here (as is the entire report for readers with a weekend to kill) and the NYT description is here.

The evaluation is a regression discontinuity design that relies on the use of a deterministic rule based on a single index to assign reading first grants. Of course, that means that the impacts really apply only to schools near the discontinuity rather than all schools, though this fact appears not to be (amazingly) mentioned in the executive summary. The overall impacts on teacher practice are in the expected direction and the overall impacts on reading comprehension are not statistically different from zero. A subgroup analysis does find some positive and statistically significant impacts on later adopting sites.

Some remarks:

1. The study is pretty well done. I participated in one conference call about it and read some material and was impressed with the level of care. Full disclosure: as a result of that one day of activity, my name appears in the executive summary as an external advisor along with the names of a large number of my friends and acquaintances.

2. One of the four main researchers is Robin Jacob, now at Michigan. Perhaps not surprisingly, the local paper - the Ann Arbor News - reprints the NYT piece with no mention of the local angle. For reasons that are not clear to me, even their locally written stories often completely ignore the subject area expertise available at the university.

3. The NYT treats the study as estimating the mean impact of the program everywhere, even though it actually estimates the mean impact only at the study sites, which were selected not at random, to allow external validity, but because they had allocation processes that fit into the RD framework, and even though at those sites it estimates the impact only for schools near the discontinuity. It also ignores the subgroup analysis and neglects to mention who actually performed the study. Remind me, once again, why anyone takes the NYT seriously?