Idea for new Institute for Education Sciences randomized trial

They should evaluate this book:

Hot X: Algebra Exposed

Description: New York Times bestselling author Danica McKellar tackles the toughest math class yet: Algebra! In her two bestselling books, Math Doesn't Suck and Kiss My Math, actress and math genius Danica McKellar shattered the "math nerd" stereotype by showing girls how to ace middle school math-and actually feel cool while doing it! Sizzling with Danica's trademark sass and style, Hot X: Algebra Exposed tackles algebra: the most feared of all math classes and the most common roadblock to high school graduation. McKellar instantly puts her readers at ease, showing teenage girls-and anyone taking algebra-how to feel confident, get in the driver's seat, and master topics like square roots, polynomials, quadratic equations, word problems and more . . . without breaking a sweat (or a nail). Danica provides illuminating, step-by-step math lessons combined with reader favorites like personality quizzes, popular doodles, real-life testimonials, and stories from her own life, so girls feel like she's sitting right next to them. As hundreds of thousands of girls already know, Danica's irreverent, light-hearted approach opens the door to higher grades and higher test scores. Now, with Hot X: Algebra Exposed, the scary veil of algebra is finally lifted, making it understandable, relevant and maybe even a little (gasp!) fun for girls.

I am eagerly awaiting the Jonas Brothers' new book on chemistry.

Hat tip: Charlie Brown

Inside the mind of John Easton

Easton is in the process of being confirmed as the new head of the Institute for Education Sciences. I am told that he is likely to be both less dynamic and less controversial than former head Grover "Russ" Whitehurst, now at Brookings.

The remarks quoted in the Education Week piece

One thing that I would like to see as a real priority for myself is to look carefully over the last six years and ask under what circumstances, and under what conditions, are particular kinds of research strategies and methodologies most likely to give the most information.

are a bit worrisome to me.

I would focus IES entirely on two things: data collection and experimental evaluations. These are the thing that are under-produced in the broader education literature. Both are also public goods, and so there is some justification for government to produce them. Put differently, Easton should focus not on balancing the IES research portfolio but the overall research portfolio.

Hat tip: friend at big IES contractor

IES TWGs in DC

I was in DC last week for two days to attend meetings of two Technical Working Groups for evaluations being funded by the Department of Education's Institute for Education Sciences (IES). These techincal working groups include staff from the evaluation contractor and IES as well as outside experts on methods (that is my usual role) and on the subject area. The outside experiments are usually mostly academics, though they are sometimes program operators or, in the case of IES evaluations, school district or teacher union officials.

On Wednesday, I went to the TWG for the evaluation of mandatory random drug testing being done by RMC Research in cooperation with Mathematica. This evaluation is at the stage of having preliminary results (which I am sworn to secrecy about) so we discussed various statistical issues related to the analysis as well as issues of presentation and focus and some secondary analyses that it would be worthwhile to undertake. The IES page on this evaluation is here.

A very interesting issue here, which we discussed at some length, is whether the key dependent variable is any drug use or frequency of drug use. This issue might seem minor, but in fact it hinges importantly on what one sees as the point of the drug testing program. If the point is to get students down to zero use, then a dummy variable for zero use is the appropriate dependent variable to highlight. In contrast, if the point is to discourage frequent use, say by moving daily or almost daily users down to occasional weekend users, then a categorical variable measuring frequency of use becomes the primary object of interest. A dummy variable for zero use versus non-zero use completely misses changes in intensity that do not lead to abstinence.

This panel was great fun, in part because I was the only economist among the experts in attendance (some of the IES folks and the consultants are also economists). We established at the last meeting that I was the only person in the room who knew what "420" meant, which I thought was kind of amusing. In any case, the other experts on the TWG are a particularly bright and outspoken lot of psychologists and such, so the discussion was great fun and very stimulating.

On Thursday I attended the TWG for the evaluation of a treatment designed to move high performing (as measured by value added in test scores) teachers to low-performing (as measured by test score levels) schools. This evaluation is earlier in the game, as the research team is just completing a pilot study of the program in a single district, and just beginning the broader evaluation in multiple districts. Though the basic design is largely set, there was lots to talk about here as well, including how likely one thinks spillovers are from high performing teachers and how long it might take any such spillovers to show up in the data on test scores in other classrooms. This has implications for what you measure and how you measure it and also for broader issues of power.

SES != socio-economic status

I was in DC on Monday to talk about an evaluation of the SES program. SES stands for supplemental educational services. It is a program I had not even heard of before but it turns out to be quite an interesting one. Under No Child Left Behind (NCLB) schools that fail to meet their performance standards (AYP in the jargon, for adequate yearly performance as I recall) for three years running must spend some of their Title I federal funds (or an equivalent amount from other sources) on what are essentially vouchers for after-school tutoring. Districts can re-capture the vouchers by attracting students to their own after-school programs but they must offer parents a choice among all state-approved providers, which includes both non-profits (both faith-based and whatever the opposite of faith-based is) and for-profits such as Sylvan learning centers. The money is pretty big here - the providers can make around $40 per hour of tutoring provided.

One important feature of this program is that spillovers are built in, as Title I funds devoted to SES are not spent on other things. This paper, which one of our ace graduate students pointed me to, is by a student of Caroline Hoxby's at Harvard and emphasizes the spillover issue.

The program raises other interesting issues as well. If vouchers for tutoring are okay, why not vouchers for the school day itself? Is it a good idea to create a new set of private sector actors who depend on federal funding? There is, apparently, a K street lobbying organization for providers of SES services, who will now fight tooth-and-nail to keep the program in place regardless of any evaluation results that might come forth. How do parents choose among alternative tutoring providers? My sense is that the information they have to go on is modest, though it may include recommendations from other parents with students in the various programs. Much of the competition may be on convenience in terms of timing and location. What determines the "dosage" of tutoring that students who do take up the services receive? That is, some students go only once, others go all year and others somewhere in between. What factors lead to this variation? What is the optimal dosage and how does it vary among students? As always, there are more interesting research questions than time to address them.

Hat tip: Alex Resch

Reading First

The Reading First evaluation performed by Abt and MDRC (and a cast of thousands) for the Institute of Education Sciences was finally released today. The executive summary is on the IES website here (as is the entire report for readers with a weekend to kill) and the NYT description is here.

The evaluation is a regression discontinuity design that relies on the use of a deterministic rule based on a single index to assign reading first grants. Of course, that means that the impacts really apply only to schools near the discontinuity rather than all schools, though this fact appears not to be (amazingly) mentioned in the executive summary. The overall impacts on teacher practice are in the expected direction and the overall impacts on reading comprehension are not statistically different from zero. A subgroup analysis does find some positive and statistically significant impacts on later adopting sites.

Some remarks:

1. The study is pretty well done. I participated in one conference call about it and read some material and was impressed with the level of care. Full disclosure: as a result of that one day of activity, my name appears in the executive summary as an external advisor along with the names of a large number of my friends and acquaintances.

2. One of the four main researchers is Robin Jacob, now at Michigan. Perhaps not surprisingly, the local paper - the Ann Arbor News - reprints the NYT piece with no mention of the local angle. For reasons that are not clear to me, even their locally written stories often completely ignore the subject area expertise available at the university.

3. The NYT treats the study as estimating the mean impact of the program everywhere, even though it actually estimates the mean impact only at the study sites, which were selected not at random, to allow external validity, but because they had allocation processes that fit into the RD framework, and even though at those sites it estimates the impact only for schools near the discontinuity. It also ignores the subgroup analysis and neglects to mention who actually performed the study. Remind me, once again, why anyone takes the NYT seriously?